2023-04-13 02:53:30,792 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc 2023-04-13 02:53:30,806 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-13 02:53:30,823 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-13 02:53:30,823 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25, deleteOnExit=true 2023-04-13 02:53:30,824 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-13 02:53:30,824 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/test.cache.data in system properties and HBase conf 2023-04-13 02:53:30,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/hadoop.tmp.dir in system properties and HBase conf 2023-04-13 02:53:30,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/hadoop.log.dir in system properties and HBase conf 2023-04-13 02:53:30,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-13 02:53:30,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-13 02:53:30,827 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-13 02:53:30,944 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-13 02:53:31,385 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-13 02:53:31,389 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-13 02:53:31,389 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-13 02:53:31,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-13 02:53:31,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 02:53:31,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-13 02:53:31,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-13 02:53:31,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 02:53:31,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 02:53:31,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-13 02:53:31,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/nfs.dump.dir in system properties and HBase conf 2023-04-13 02:53:31,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/java.io.tmpdir in system properties and HBase conf 2023-04-13 02:53:31,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 02:53:31,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-13 02:53:31,393 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-13 02:53:31,969 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 02:53:31,974 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 02:53:32,270 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-13 02:53:32,458 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-13 02:53:32,475 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 02:53:32,513 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 02:53:32,544 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/java.io.tmpdir/Jetty_localhost_35975_hdfs____.oft735/webapp 2023-04-13 02:53:32,685 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35975 2023-04-13 02:53:32,747 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 02:53:32,747 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 02:53:33,187 WARN [Listener at localhost/37981] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 02:53:33,317 WARN [Listener at localhost/37981] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-13 02:53:33,339 WARN [Listener at localhost/37981] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 02:53:33,347 INFO [Listener at localhost/37981] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 02:53:33,352 INFO [Listener at localhost/37981] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/java.io.tmpdir/Jetty_localhost_46261_datanode____scu4nm/webapp 2023-04-13 02:53:33,458 INFO [Listener at localhost/37981] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46261 2023-04-13 02:53:33,846 WARN [Listener at localhost/34379] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 02:53:34,445 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbc12f9ee68189052: Processing first storage report for DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6 from datanode c27897a0-e461-4578-8075-79a476069002 2023-04-13 02:53:34,447 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbc12f9ee68189052: from storage DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6 node DatanodeRegistration(127.0.0.1:41801, datanodeUuid=c27897a0-e461-4578-8075-79a476069002, infoPort=39591, infoSecurePort=0, ipcPort=34379, storageInfo=lv=-57;cid=testClusterID;nsid=1579110615;c=1681354412046), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-13 02:53:34,447 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbc12f9ee68189052: Processing first storage report for DS-56d9f01e-a878-46d3-bf2a-1ce9dd4d9be9 from datanode c27897a0-e461-4578-8075-79a476069002 2023-04-13 02:53:34,448 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbc12f9ee68189052: from storage DS-56d9f01e-a878-46d3-bf2a-1ce9dd4d9be9 node DatanodeRegistration(127.0.0.1:41801, datanodeUuid=c27897a0-e461-4578-8075-79a476069002, infoPort=39591, infoSecurePort=0, ipcPort=34379, storageInfo=lv=-57;cid=testClusterID;nsid=1579110615;c=1681354412046), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-13 02:53:34,456 DEBUG [Listener at localhost/34379] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc 2023-04-13 02:53:34,538 INFO [Listener at localhost/34379] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25/zookeeper_0, clientPort=62948, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-13 02:53:34,553 INFO [Listener at localhost/34379] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62948 2023-04-13 02:53:34,562 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:34,565 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:35,223 INFO [Listener at localhost/34379] util.FSUtils(471): Created version file at hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55 with version=8 2023-04-13 02:53:35,223 INFO [Listener at localhost/34379] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/hbase-staging 2023-04-13 02:53:35,516 INFO [Listener at localhost/34379] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-13 02:53:35,971 INFO [Listener at localhost/34379] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 02:53:36,002 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,003 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,003 INFO [Listener at localhost/34379] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 02:53:36,003 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,004 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 02:53:36,144 INFO [Listener at localhost/34379] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 02:53:36,223 DEBUG [Listener at localhost/34379] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-13 02:53:36,336 INFO [Listener at localhost/34379] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37305 2023-04-13 02:53:36,346 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:36,348 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:36,373 INFO [Listener at localhost/34379] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37305 connecting to ZooKeeper ensemble=127.0.0.1:62948 2023-04-13 02:53:36,412 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:373050x0, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 02:53:36,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37305-0x1017b7043800000 connected 2023-04-13 02:53:36,439 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 02:53:36,440 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 02:53:36,443 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 02:53:36,450 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37305 2023-04-13 02:53:36,451 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37305 2023-04-13 02:53:36,451 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37305 2023-04-13 02:53:36,452 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37305 2023-04-13 02:53:36,452 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37305 2023-04-13 02:53:36,458 INFO [Listener at localhost/34379] master.HMaster(444): hbase.rootdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55, hbase.cluster.distributed=false 2023-04-13 02:53:36,534 INFO [Listener at localhost/34379] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 02:53:36,535 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,535 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,535 INFO [Listener at localhost/34379] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 02:53:36,535 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 02:53:36,535 INFO [Listener at localhost/34379] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 02:53:36,540 INFO [Listener at localhost/34379] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 02:53:36,544 INFO [Listener at localhost/34379] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34919 2023-04-13 02:53:36,546 INFO [Listener at localhost/34379] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-13 02:53:36,552 DEBUG [Listener at localhost/34379] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-13 02:53:36,552 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:36,555 INFO [Listener at localhost/34379] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:36,557 INFO [Listener at localhost/34379] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34919 connecting to ZooKeeper ensemble=127.0.0.1:62948 2023-04-13 02:53:36,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:349190x0, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 02:53:36,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34919-0x1017b7043800001 connected 2023-04-13 02:53:36,561 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 02:53:36,562 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 02:53:36,563 DEBUG [Listener at localhost/34379] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 02:53:36,564 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34919 2023-04-13 02:53:36,565 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34919 2023-04-13 02:53:36,568 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34919 2023-04-13 02:53:36,568 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34919 2023-04-13 02:53:36,569 DEBUG [Listener at localhost/34379] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34919 2023-04-13 02:53:36,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:36,580 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 02:53:36,581 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:36,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 02:53:36,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 02:53:36,607 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:36,608 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 02:53:36,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37305,1681354415362 from backup master directory 2023-04-13 02:53:36,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 02:53:36,614 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:36,614 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 02:53:36,615 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 02:53:36,615 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:36,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-13 02:53:36,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-13 02:53:36,725 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/hbase.id with ID: e7557fdc-c2cc-488f-8e17-83eb4f45a529 2023-04-13 02:53:36,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 02:53:36,786 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:36,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x01d21cc5 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:36,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58799cb1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:36,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:53:36,890 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-13 02:53:36,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-13 02:53:36,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-13 02:53:36,910 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:53:36,915 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:53:36,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 02:53:36,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store-tmp 2023-04-13 02:53:36,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:36,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 02:53:36,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:53:36,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:53:36,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 02:53:36,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:53:36,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:53:36,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 02:53:36,990 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/WALs/jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:37,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37305%2C1681354415362, suffix=, logDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/WALs/jenkins-hbase4.apache.org,37305,1681354415362, archiveDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/oldWALs, maxLogs=10 2023-04-13 02:53:37,072 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK] 2023-04-13 02:53:37,080 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:53:37,143 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/WALs/jenkins-hbase4.apache.org,37305,1681354415362/jenkins-hbase4.apache.org%2C37305%2C1681354415362.1681354417026 2023-04-13 02:53:37,144 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK]] 2023-04-13 02:53:37,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:53:37,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:37,148 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,209 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,217 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-13 02:53:37,248 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-13 02:53:37,263 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:37,269 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 02:53:37,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:53:37,294 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61180727, jitterRate=-0.08833612501621246}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 02:53:37,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 02:53:37,295 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-13 02:53:37,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-13 02:53:37,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-13 02:53:37,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-13 02:53:37,329 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-13 02:53:37,373 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 44 msec 2023-04-13 02:53:37,374 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-13 02:53:37,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-13 02:53:37,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-13 02:53:37,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-13 02:53:37,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-13 02:53:37,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-13 02:53:37,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-13 02:53:37,448 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-13 02:53:37,451 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:37,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-13 02:53:37,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-13 02:53:37,465 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-13 02:53:37,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 02:53:37,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 02:53:37,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:37,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,37305,1681354415362, sessionid=0x1017b7043800000, setting cluster-up flag (Was=false) 2023-04-13 02:53:37,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:37,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-13 02:53:37,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:37,501 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:37,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-13 02:53:37,509 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:37,513 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.hbase-snapshot/.tmp 2023-04-13 02:53:37,574 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(952): ClusterId : e7557fdc-c2cc-488f-8e17-83eb4f45a529 2023-04-13 02:53:37,580 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-13 02:53:37,586 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-13 02:53:37,586 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-13 02:53:37,591 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-13 02:53:37,592 DEBUG [RS:0;jenkins-hbase4:34919] zookeeper.ReadOnlyZKClient(139): Connect 0x7ac91733 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:37,603 DEBUG [RS:0;jenkins-hbase4:34919] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@740ee61a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:37,604 DEBUG [RS:0;jenkins-hbase4:34919] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70317c43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 02:53:37,637 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34919 2023-04-13 02:53:37,642 INFO [RS:0;jenkins-hbase4:34919] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-13 02:53:37,642 INFO [RS:0;jenkins-hbase4:34919] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-13 02:53:37,642 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1023): About to register with Master. 2023-04-13 02:53:37,646 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37305,1681354415362 with isa=jenkins-hbase4.apache.org/172.31.14.131:34919, startcode=1681354416534 2023-04-13 02:53:37,658 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-13 02:53:37,669 DEBUG [RS:0;jenkins-hbase4:34919] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-13 02:53:37,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 02:53:37,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 02:53:37,669 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 02:53:37,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 02:53:37,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-13 02:53:37,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 02:53:37,670 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,684 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681354447683 2023-04-13 02:53:37,686 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-13 02:53:37,689 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 02:53:37,690 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-13 02:53:37,698 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 02:53:37,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-13 02:53:37,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-13 02:53:37,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-13 02:53:37,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-13 02:53:37,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-13 02:53:37,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,722 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-13 02:53:37,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-13 02:53:37,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-13 02:53:37,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-13 02:53:37,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-13 02:53:37,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681354417735,5,FailOnTimeoutGroup] 2023-04-13 02:53:37,738 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39981, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-13 02:53:37,738 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681354417736,5,FailOnTimeoutGroup] 2023-04-13 02:53:37,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,739 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-13 02:53:37,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,776 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55 2023-04-13 02:53:37,776 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37981 2023-04-13 02:53:37,776 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-13 02:53:37,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 02:53:37,782 DEBUG [RS:0;jenkins-hbase4:34919] zookeeper.ZKUtil(162): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,782 WARN [RS:0;jenkins-hbase4:34919] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 02:53:37,783 INFO [RS:0;jenkins-hbase4:34919] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 02:53:37,783 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/WALs/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,784 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34919,1681354416534] 2023-04-13 02:53:37,793 DEBUG [RS:0;jenkins-hbase4:34919] zookeeper.ZKUtil(162): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,806 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-13 02:53:37,818 INFO [RS:0;jenkins-hbase4:34919] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-13 02:53:37,840 INFO [RS:0;jenkins-hbase4:34919] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-13 02:53:37,844 INFO [RS:0;jenkins-hbase4:34919] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-13 02:53:37,844 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,846 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-13 02:53:37,853 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,853 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,854 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,854 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,854 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,854 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,855 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 02:53:37,855 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,855 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,855 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,855 DEBUG [RS:0;jenkins-hbase4:34919] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 02:53:37,856 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,857 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,857 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,873 INFO [RS:0;jenkins-hbase4:34919] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-13 02:53:37,876 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34919,1681354416534-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:37,892 INFO [RS:0;jenkins-hbase4:34919] regionserver.Replication(203): jenkins-hbase4.apache.org,34919,1681354416534 started 2023-04-13 02:53:37,892 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34919,1681354416534, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34919, sessionid=0x1017b7043800001 2023-04-13 02:53:37,893 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-13 02:53:37,893 DEBUG [RS:0;jenkins-hbase4:34919] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,893 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34919,1681354416534' 2023-04-13 02:53:37,893 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:37,894 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:37,894 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-13 02:53:37,894 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-13 02:53:37,895 DEBUG [RS:0;jenkins-hbase4:34919] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:37,895 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34919,1681354416534' 2023-04-13 02:53:37,895 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-13 02:53:37,895 DEBUG [RS:0;jenkins-hbase4:34919] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-13 02:53:37,896 DEBUG [RS:0;jenkins-hbase4:34919] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-13 02:53:37,896 INFO [RS:0;jenkins-hbase4:34919] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-13 02:53:37,896 INFO [RS:0;jenkins-hbase4:34919] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-13 02:53:38,008 INFO [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34919%2C1681354416534, suffix=, logDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/WALs/jenkins-hbase4.apache.org,34919,1681354416534, archiveDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/oldWALs, maxLogs=32 2023-04-13 02:53:38,031 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK] 2023-04-13 02:53:38,035 INFO [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/WALs/jenkins-hbase4.apache.org,34919,1681354416534/jenkins-hbase4.apache.org%2C34919%2C1681354416534.1681354418011 2023-04-13 02:53:38,035 DEBUG [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK]] 2023-04-13 02:53:38,155 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 02:53:38,157 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 02:53:38,158 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55 2023-04-13 02:53:38,175 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:38,178 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 02:53:38,182 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/info 2023-04-13 02:53:38,183 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 02:53:38,184 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,184 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 02:53:38,187 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/rep_barrier 2023-04-13 02:53:38,188 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 02:53:38,189 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,189 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 02:53:38,192 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/table 2023-04-13 02:53:38,192 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 02:53:38,193 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,195 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740 2023-04-13 02:53:38,196 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740 2023-04-13 02:53:38,200 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:53:38,202 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 02:53:38,207 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:53:38,208 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70270757, jitterRate=0.04711587727069855}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:53:38,208 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 02:53:38,208 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 02:53:38,208 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 02:53:38,208 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 02:53:38,208 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 02:53:38,209 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 02:53:38,211 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 02:53:38,211 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 02:53:38,216 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 02:53:38,216 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-13 02:53:38,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-13 02:53:38,238 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-13 02:53:38,240 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-13 02:53:38,392 DEBUG [jenkins-hbase4:37305] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-13 02:53:38,396 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34919,1681354416534, state=OPENING 2023-04-13 02:53:38,403 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-13 02:53:38,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:38,405 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 02:53:38,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:53:38,598 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:38,600 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-13 02:53:38,605 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-13 02:53:38,617 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-13 02:53:38,617 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 02:53:38,622 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34919%2C1681354416534.meta, suffix=.meta, logDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/WALs/jenkins-hbase4.apache.org,34919,1681354416534, archiveDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/oldWALs, maxLogs=32 2023-04-13 02:53:38,644 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK] 2023-04-13 02:53:38,649 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/WALs/jenkins-hbase4.apache.org,34919,1681354416534/jenkins-hbase4.apache.org%2C34919%2C1681354416534.meta.1681354418624.meta 2023-04-13 02:53:38,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41801,DS-f2473958-7d4b-45dc-b144-d8a5d7da8df6,DISK]] 2023-04-13 02:53:38,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:53:38,653 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-13 02:53:38,670 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-13 02:53:38,676 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-13 02:53:38,682 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-13 02:53:38,682 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:38,682 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-13 02:53:38,682 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-13 02:53:38,685 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 02:53:38,687 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/info 2023-04-13 02:53:38,687 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/info 2023-04-13 02:53:38,688 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 02:53:38,689 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,689 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 02:53:38,690 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/rep_barrier 2023-04-13 02:53:38,691 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/rep_barrier 2023-04-13 02:53:38,691 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 02:53:38,692 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,692 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 02:53:38,694 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/table 2023-04-13 02:53:38,694 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/table 2023-04-13 02:53:38,694 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 02:53:38,695 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:38,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740 2023-04-13 02:53:38,699 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740 2023-04-13 02:53:38,703 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:53:38,705 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 02:53:38,707 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74301898, jitterRate=0.10718455910682678}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:53:38,707 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 02:53:38,716 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681354418590 2023-04-13 02:53:38,737 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-13 02:53:38,738 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-13 02:53:38,739 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34919,1681354416534, state=OPEN 2023-04-13 02:53:38,741 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-13 02:53:38,742 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 02:53:38,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-13 02:53:38,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34919,1681354416534 in 334 msec 2023-04-13 02:53:38,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-13 02:53:38,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 523 msec 2023-04-13 02:53:38,758 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1860 sec 2023-04-13 02:53:38,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681354418758, completionTime=-1 2023-04-13 02:53:38,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-13 02:53:38,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-13 02:53:38,835 DEBUG [hconnection-0x12cad558-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:38,840 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:38,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-13 02:53:38,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681354478860 2023-04-13 02:53:38,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681354538860 2023-04-13 02:53:38,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 100 msec 2023-04-13 02:53:38,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37305,1681354415362-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:38,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37305,1681354415362-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:38,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37305,1681354415362-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:38,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37305, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:38,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-13 02:53:38,895 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-13 02:53:38,906 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-13 02:53:38,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 02:53:38,919 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-13 02:53:38,922 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:53:38,925 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:53:38,947 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/hbase/namespace/1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:38,949 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/hbase/namespace/1926898cb36187246d86e87ddb72126d empty. 2023-04-13 02:53:38,950 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/hbase/namespace/1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:38,950 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-13 02:53:39,375 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-13 02:53:39,377 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1926898cb36187246d86e87ddb72126d, NAME => 'hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:53:39,792 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:39,793 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 1926898cb36187246d86e87ddb72126d, disabling compactions & flushes 2023-04-13 02:53:39,793 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:39,793 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:39,793 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. after waiting 0 ms 2023-04-13 02:53:39,793 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:39,793 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:39,793 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 1926898cb36187246d86e87ddb72126d: 2023-04-13 02:53:39,799 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:53:39,814 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681354419801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354419801"}]},"ts":"1681354419801"} 2023-04-13 02:53:39,843 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:53:39,845 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:53:39,849 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354419845"}]},"ts":"1681354419845"} 2023-04-13 02:53:39,854 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-13 02:53:39,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1926898cb36187246d86e87ddb72126d, ASSIGN}] 2023-04-13 02:53:39,866 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1926898cb36187246d86e87ddb72126d, ASSIGN 2023-04-13 02:53:39,868 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1926898cb36187246d86e87ddb72126d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:53:40,019 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1926898cb36187246d86e87ddb72126d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:40,019 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681354420019"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354420019"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354420019"}]},"ts":"1681354420019"} 2023-04-13 02:53:40,024 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1926898cb36187246d86e87ddb72126d, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:53:40,183 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:40,184 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1926898cb36187246d86e87ddb72126d, NAME => 'hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:53:40,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:40,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,185 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,187 INFO [StoreOpener-1926898cb36187246d86e87ddb72126d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,189 DEBUG [StoreOpener-1926898cb36187246d86e87ddb72126d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/info 2023-04-13 02:53:40,189 DEBUG [StoreOpener-1926898cb36187246d86e87ddb72126d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/info 2023-04-13 02:53:40,190 INFO [StoreOpener-1926898cb36187246d86e87ddb72126d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1926898cb36187246d86e87ddb72126d columnFamilyName info 2023-04-13 02:53:40,191 INFO [StoreOpener-1926898cb36187246d86e87ddb72126d-1] regionserver.HStore(310): Store=1926898cb36187246d86e87ddb72126d/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:40,192 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,193 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,198 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1926898cb36187246d86e87ddb72126d 2023-04-13 02:53:40,201 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:53:40,202 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1926898cb36187246d86e87ddb72126d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64102257, jitterRate=-0.04480193555355072}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 02:53:40,202 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1926898cb36187246d86e87ddb72126d: 2023-04-13 02:53:40,204 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d., pid=6, masterSystemTime=1681354420177 2023-04-13 02:53:40,208 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:40,208 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:53:40,210 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1926898cb36187246d86e87ddb72126d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:40,210 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681354420209"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354420209"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354420209"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354420209"}]},"ts":"1681354420209"} 2023-04-13 02:53:40,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-13 02:53:40,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1926898cb36187246d86e87ddb72126d, server=jenkins-hbase4.apache.org,34919,1681354416534 in 191 msec 2023-04-13 02:53:40,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-13 02:53:40,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1926898cb36187246d86e87ddb72126d, ASSIGN in 356 msec 2023-04-13 02:53:40,224 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:53:40,224 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354420224"}]},"ts":"1681354420224"} 2023-04-13 02:53:40,227 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-13 02:53:40,231 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:53:40,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3230 sec 2023-04-13 02:53:40,323 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-13 02:53:40,325 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-13 02:53:40,325 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:53:40,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-13 02:53:40,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 02:53:40,398 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 40 msec 2023-04-13 02:53:40,411 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-13 02:53:40,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 02:53:40,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-04-13 02:53:40,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-13 02:53:40,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-13 02:53:40,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.826sec 2023-04-13 02:53:40,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-13 02:53:40,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-13 02:53:40,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-13 02:53:40,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37305,1681354415362-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-13 02:53:40,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37305,1681354415362-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-13 02:53:40,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-13 02:53:40,481 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x45b2cf6e to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:40,485 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bc25e59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:40,498 DEBUG [hconnection-0x5d5e6c96-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:40,509 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:40,520 INFO [Listener at localhost/34379] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:53:40,543 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=171, AvailableMemoryMB=1705 2023-04-13 02:53:40,561 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:53:40,564 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:53:40,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:53:40,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:53:40,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:53:40,582 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:53:40,584 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:53:40,587 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-13 02:53:40,588 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e empty. 2023-04-13 02:53:40,590 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,590 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:53:40,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 02:53:40,608 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:53:40,610 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 23777bf909bb0929ddad8e9cb44ee33e, NAME => 'TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:53:40,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:40,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 23777bf909bb0929ddad8e9cb44ee33e, disabling compactions & flushes 2023-04-13 02:53:40,629 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. after waiting 0 ms 2023-04-13 02:53:40,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,629 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:40,637 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:53:40,640 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354420639"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354420639"}]},"ts":"1681354420639"} 2023-04-13 02:53:40,643 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:53:40,645 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:53:40,645 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354420645"}]},"ts":"1681354420645"} 2023-04-13 02:53:40,648 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:53:40,653 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, ASSIGN}] 2023-04-13 02:53:40,656 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, ASSIGN 2023-04-13 02:53:40,658 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:53:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 02:53:40,809 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=23777bf909bb0929ddad8e9cb44ee33e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:40,809 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354420809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354420809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354420809"}]},"ts":"1681354420809"} 2023-04-13 02:53:40,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:53:40,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 02:53:40,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 23777bf909bb0929ddad8e9cb44ee33e, NAME => 'TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:53:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:53:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,980 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,982 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A 2023-04-13 02:53:40,983 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A 2023-04-13 02:53:40,984 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:53:40,985 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 23777bf909bb0929ddad8e9cb44ee33e columnFamilyName A 2023-04-13 02:53:40,986 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(310): Store=23777bf909bb0929ddad8e9cb44ee33e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:40,986 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,989 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B 2023-04-13 02:53:40,989 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B 2023-04-13 02:53:40,989 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:53:40,990 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 23777bf909bb0929ddad8e9cb44ee33e columnFamilyName B 2023-04-13 02:53:40,991 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(310): Store=23777bf909bb0929ddad8e9cb44ee33e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:40,991 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,993 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C 2023-04-13 02:53:40,993 DEBUG [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C 2023-04-13 02:53:40,993 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:53:40,994 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 23777bf909bb0929ddad8e9cb44ee33e columnFamilyName C 2023-04-13 02:53:40,995 INFO [StoreOpener-23777bf909bb0929ddad8e9cb44ee33e-1] regionserver.HStore(310): Store=23777bf909bb0929ddad8e9cb44ee33e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:53:40,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:40,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:40,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:41,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:53:41,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:41,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:53:41,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 23777bf909bb0929ddad8e9cb44ee33e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68370219, jitterRate=0.01879565417766571}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:53:41,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:41,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., pid=11, masterSystemTime=1681354420966 2023-04-13 02:53:41,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:41,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:41,013 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=23777bf909bb0929ddad8e9cb44ee33e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:41,014 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354421013"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354421013"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354421013"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354421013"}]},"ts":"1681354421013"} 2023-04-13 02:53:41,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-13 02:53:41,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 in 203 msec 2023-04-13 02:53:41,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-13 02:53:41,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, ASSIGN in 366 msec 2023-04-13 02:53:41,024 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:53:41,024 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354421024"}]},"ts":"1681354421024"} 2023-04-13 02:53:41,027 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:53:41,030 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:53:41,032 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 453 msec 2023-04-13 02:53:41,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 02:53:41,213 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-13 02:53:41,219 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x5d4f988c to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,235 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23162110, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,238 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,242 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,245 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:53:41,248 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:53:41,257 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x72ea4a65 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,261 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e82daee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,262 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1645b6b0 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,269 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e8196e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,269 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x7b52aa42 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,273 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e2005d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,274 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1710407d to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,278 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57de6c0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,278 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x7bc6d729 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,282 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f5a5e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,282 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x00375c9d to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,286 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64804459, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,287 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x2ec2fae8 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,333 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142b9819, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,334 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x27685d29 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,338 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbe28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,339 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x5979620a to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:53:41,344 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9ded19, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:53:41,351 DEBUG [hconnection-0x2a81f259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,357 DEBUG [hconnection-0x567266f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,360 DEBUG [hconnection-0x7a95e8d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,360 DEBUG [hconnection-0x12fded7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,359 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,364 DEBUG [hconnection-0x6d40587f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,364 DEBUG [hconnection-0x37e8943a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,367 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,372 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,377 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,378 DEBUG [hconnection-0x5f9b9bf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:53:41,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:41,389 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,394 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,394 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,397 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,410 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,411 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:53:41,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:41,448 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:41,448 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:41,449 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:41,449 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:41,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:41,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:41,452 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:41,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:41,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:41,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:41,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:41,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:41,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:41,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:41,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:41,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:41,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:41,468 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:41,472 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:41,476 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:53:41,489 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:41,490 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:41,491 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:41,491 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:41,491 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:41,491 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:41,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:41,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354421565, max timestamp=1681354421587Num uniques 859; 2023-04-13 02:53:41,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:41,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354421565, max timestamp=1681354421587Num uniques 859; 2023-04-13 02:53:41,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:41,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354421565, max timestamp=1681354421587Num uniques 859; 2023-04-13 02:53:41,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:41,674 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/d1c7b314755b497297f7e430f1286862 2023-04-13 02:53:41,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:41,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354421653, max timestamp=1681354421676Num uniques 1068; 2023-04-13 02:53:41,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-13 02:53:41,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9977, totalHeapSize=22912, min timestamp=1681354421653, max timestamp=1681354421678Num uniques 1018; 2023-04-13 02:53:41,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46617, new segment data size=9977, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 02:53:41,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354481671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:41,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354421653, max timestamp=1681354421678Num uniques 1068; 2023-04-13 02:53:41,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-13 02:53:41,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681354421780, max timestamp=1681354421793Num uniques 1077; 2023-04-13 02:53:41,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 02:53:41,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354421780, max timestamp=1681354421793Num uniques 1009; 2023-04-13 02:53:41,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 02:53:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354421780, max timestamp=1681354421793Num uniques 1009; 2023-04-13 02:53:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 02:53:41,854 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/89304eca7d6b4cd0af7c3f7ff30362dd 2023-04-13 02:53:41,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354421851, max timestamp=1681354421874Num uniques 1009; 2023-04-13 02:53:41,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354421851, max timestamp=1681354421878Num uniques 1009; 2023-04-13 02:53:41,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-13 02:53:41,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-13 02:53:41,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354421851, max timestamp=1681354421870Num uniques 1036; 2023-04-13 02:53:41,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-13 02:53:41,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354421972, max timestamp=1681354421982Num uniques 1009; 2023-04-13 02:53:41,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354421972, max timestamp=1681354421982Num uniques 1045; 2023-04-13 02:53:41,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-04-13 02:53:41,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354421972, max timestamp=1681354421982Num uniques 1045; 2023-04-13 02:53:41,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 02:53:41,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354422005, max timestamp=1681354422021Num uniques 1054; 2023-04-13 02:53:42,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354422005, max timestamp=1681354422021Num uniques 1054; 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354422005, max timestamp=1681354422021Num uniques 1054; 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 02:53:42,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 02:53:42,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354482020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681354422048, max timestamp=1681354422058Num uniques 1013; 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681354422048, max timestamp=1681354422062Num uniques 1013; 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681354422048, max timestamp=1681354422062Num uniques 1013; 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 02:53:42,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 02:53:42,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681354422082, max timestamp=1681354422099Num uniques 1022; 2023-04-13 02:53:42,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 02:53:42,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681354422082, max timestamp=1681354422099Num uniques 1022; 2023-04-13 02:53:42,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 02:53:42,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681354422082, max timestamp=1681354422099Num uniques 1022; 2023-04-13 02:53:42,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 02:53:42,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681354422126, max timestamp=1681354422142Num uniques 1031; 2023-04-13 02:53:42,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681354422126, max timestamp=1681354422142Num uniques 1031; 2023-04-13 02:53:42,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:53:42,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:53:42,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681354422126, max timestamp=1681354422142Num uniques 1031; 2023-04-13 02:53:42,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:53:42,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681354422153, max timestamp=1681354422181Num uniques 1040; 2023-04-13 02:53:42,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-13 02:53:42,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681354422153, max timestamp=1681354422181Num uniques 1040; 2023-04-13 02:53:42,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-13 02:53:42,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354422173, max timestamp=1681354422181Num uniques 1040; 2023-04-13 02:53:42,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1681354422192, max timestamp=1681354422237Num uniques 1149; 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1681354422192, max timestamp=1681354422237Num uniques 1009; 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=15984, suffix heap size=147464,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=349 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1681354422192, max timestamp=1681354422237Num uniques 1149; 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15984, suffix heap size=132904,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=349 2023-04-13 02:53:42,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=15984, suffix heap size=147464,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=349 2023-04-13 02:53:42,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354482269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354482272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354482276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354482276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681354422265, max timestamp=1681354422275Num uniques 1208; 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681354422265, max timestamp=1681354422275Num uniques 1208; 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681354422265, max timestamp=1681354422275Num uniques 1208; 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=7230, suffix heap size=153600,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=158 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=7230, suffix heap size=153600,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=158 2023-04-13 02:53:42,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=7230, suffix heap size=153600,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=158 2023-04-13 02:53:42,338 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/8ecdc06b82404d80b78712fff33f1a0f 2023-04-13 02:53:42,355 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/d1c7b314755b497297f7e430f1286862 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862 2023-04-13 02:53:42,370 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862, entries=100, sequenceid=8, filesize=9.3 K 2023-04-13 02:53:42,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681354422326, max timestamp=1681354422370Num uniques 1017; 2023-04-13 02:53:42,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-13 02:53:42,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681354422326, max timestamp=1681354422370Num uniques 1017; 2023-04-13 02:53:42,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681354422326, max timestamp=1681354422370Num uniques 1017; 2023-04-13 02:53:42,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-13 02:53:42,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-13 02:53:42,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/89304eca7d6b4cd0af7c3f7ff30362dd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd 2023-04-13 02:53:42,410 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd, entries=100, sequenceid=8, filesize=9.3 K 2023-04-13 02:53:42,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/8ecdc06b82404d80b78712fff33f1a0f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f 2023-04-13 02:53:42,430 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f, entries=100, sequenceid=8, filesize=9.3 K 2023-04-13 02:53:42,442 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for 23777bf909bb0929ddad8e9cb44ee33e in 966ms, sequenceid=8, compaction requested=false 2023-04-13 02:53:42,444 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 02:53:42,447 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:42,447 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:42,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:42,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=67.09 KB heapSize=180.33 KB 2023-04-13 02:53:42,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:42,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:42,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:42,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:42,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:42,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:42,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:42,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,455 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:42,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:42,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1152b627[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:42,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,464 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:42,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:42,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:42,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:42,473 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:42,473 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,473 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:42,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:42,486 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:42,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:42,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:42,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:42,486 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:42,490 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,490 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,493 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,494 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,515 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,516 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:42,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:42,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:42,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:42,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:42,520 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:42,520 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:42,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:42,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354482585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,620 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:42,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354482589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354482589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354482595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354482616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:42,644 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:42,644 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:42,647 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:42,647 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:42,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:42,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:42,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:42,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:42,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:42,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:42,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:42,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:42,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:42,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:42,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:42,659 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:42,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d49951e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:42,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:42,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:42,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:42,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:42,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:42,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:42,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:42,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:42,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:42,687 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:42,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:42,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:42,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:42,687 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:42,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,697 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,700 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,710 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:42,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:42,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,715 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:42,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:42,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:42,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:42,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:42,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,724 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354482724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354482729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354482730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354482730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354482730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:42,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:42,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:42,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:42,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:42,846 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:42,846 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:42,848 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:42,848 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:42,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,851 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:42,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:42,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:42,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:42,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:42,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:42,857 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:42,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:42,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:42,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:42,861 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:42,861 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:42,861 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:42,861 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:42,866 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,866 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,866 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:42,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1e08b7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:42,871 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:42,871 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,873 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:42,873 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:42,874 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:42,873 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,874 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:42,884 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:42,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:42,888 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:42,888 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:42,889 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:42,889 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:42,889 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:42,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:42,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:42,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,923 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:42,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:42,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:42,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:42,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:42,925 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:42,925 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:42,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e77d80f620b9410e8e4e3d2cd08915b3 2023-04-13 02:53:42,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354482947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354482957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354482958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354482958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:42,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:42,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354482958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/caa6b11ad3794397b56094efbac183ab 2023-04-13 02:53:43,025 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:43,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:43,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:43,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:43,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:43,065 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:43,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,088 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:43,089 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:43,089 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,089 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,091 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,091 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,093 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,093 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:43,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:43,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:43,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:43,120 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:43,120 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:43,120 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:43,120 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:43,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:43,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:43,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:43,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:43,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22ba535[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:43,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:43,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:43,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:43,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:43,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:43,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:43,184 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:43,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:43,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:43,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:43,184 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:43,194 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,194 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,194 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,197 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:43,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:43,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:43,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:43,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354483257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354483268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354483269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354483300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354483308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:43,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:43,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:43,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:43,346 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:43,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:43,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:43,349 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,349 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,351 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,351 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:43,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:43,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:43,361 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:43,361 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:43,364 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,364 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:43,364 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:43,364 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:43,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:43,369 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:43,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cc8d9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:43,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:43,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:43,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:43,378 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,378 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:43,385 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:43,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:43,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:43,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:43,385 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:43,389 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,389 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,394 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,402 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:43,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,417 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:43,417 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:43,520 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:43,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:43,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:43,548 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:43,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:43,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:43,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,553 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,553 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:43,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:43,556 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:43,561 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:43,561 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,561 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:43,562 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:43,562 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,564 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:43,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:43,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:43,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ac5c6b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:43,572 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:43,572 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,574 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:43,575 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,575 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:43,580 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:43,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:43,588 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:43,588 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:43,588 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:43,588 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:43,588 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:43,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,598 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,598 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,598 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,598 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,598 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,599 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,599 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,599 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:43,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:43,599 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,605 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:43,605 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/6f29182a1bd443e88196dc4b6b25f617 2023-04-13 02:53:43,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e77d80f620b9410e8e4e3d2cd08915b3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3 2023-04-13 02:53:43,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3, entries=250, sequenceid=240, filesize=16.4 K 2023-04-13 02:53:43,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/caa6b11ad3794397b56094efbac183ab as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab 2023-04-13 02:53:43,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab, entries=150, sequenceid=240, filesize=11.8 K 2023-04-13 02:53:43,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/6f29182a1bd443e88196dc4b6b25f617 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617 2023-04-13 02:53:43,705 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:43,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:43,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:43,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:43,719 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:43,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:43,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:43,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,736 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:43,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:43,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:43,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:43,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:43,755 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,756 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:43,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:43,756 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:43,756 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:43,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:43,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,762 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:43,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d6da799[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:43,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:43,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,775 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:43,776 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:43,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:43,778 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354483782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617, entries=150, sequenceid=240, filesize=11.8 K 2023-04-13 02:53:43,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354483780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~180.28 KB/184608, currentSize=114.05 KB/116790 for 23777bf909bb0929ddad8e9cb44ee33e in 1336ms, sequenceid=240, compaction requested=false 2023-04-13 02:53:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:43,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:43,791 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:43,791 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:43,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354422529, max timestamp=1681354423817Num uniques 859; 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354422529, max timestamp=1681354423817Num uniques 859; 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354422529, max timestamp=1681354423817Num uniques 859; 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:43,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:43,824 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:43,824 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:43,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:43,831 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:43,925 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:43,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:43,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:43,934 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:43,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:43,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:43,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,942 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:43,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:43,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:43,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:43,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:43,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:43,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:43,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354423878, max timestamp=1681354423944Num uniques 859; 2023-04-13 02:53:43,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:43,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354423878, max timestamp=1681354423944Num uniques 859; 2023-04-13 02:53:43,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:43,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354423878, max timestamp=1681354423944Num uniques 859; 2023-04-13 02:53:43,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:43,962 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,962 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:43,962 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:43,962 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:43,965 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:43,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4230ab0c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:43,968 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:43,968 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:43,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:43,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:43,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:43,971 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:43,971 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,971 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:43,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:43,978 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:43,978 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:43,978 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:43,978 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:43,978 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:43,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,984 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:43,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:43,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:43,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:43,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:43,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354483992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:43,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,996 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354483995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:43,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:43,997 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:43,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354483997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,098 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:44,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:44,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354484099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:44,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354484101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354484101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:44,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:44,111 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:44,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:44,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:44,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:44,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:44,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:44,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:44,124 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:44,124 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:44,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:44,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:44,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:44,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:44,128 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,128 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:44,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22daa993[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:44,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:44,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:44,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:44,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:44,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:44,138 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:44,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:44,142 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:44,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:44,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:44,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:44,142 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:44,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,144 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,144 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:44,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,152 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:44,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,153 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:44,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,253 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:44,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:44,278 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:44,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:44,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:44,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:44,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:44,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:44,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:44,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:44,286 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,286 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:44,286 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:44,287 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:44,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:44,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4770f6a7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:44,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:44,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/7fcabd81360b4ef5995bc183ecdb1d2b 2023-04-13 02:53:44,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:44,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:44,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:44,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:44,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,305 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:44,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:44,313 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:44,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:44,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:44,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:44,313 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:44,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354484309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,324 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,322 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,322 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,330 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,316 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:53:44,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354484310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354484313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,340 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,356 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:44,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,367 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:44,367 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:44,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:44,421 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/208ff3d80e494d5c8cac5b2612b97d7e] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:53:44,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/208ff3d80e494d5c8cac5b2612b97d7e 2023-04-13 02:53:44,467 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:44,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:44,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:44,499 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:44,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:44,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:44,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,507 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,507 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:44,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:44,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:44,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:44,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:44,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/849d7f335c7c40cfa1c4b603635d64f1 2023-04-13 02:53:44,567 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,567 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:44,567 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:44,567 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:44,575 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,575 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,576 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:44,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3751c0bf[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:44,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:44,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,581 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:44,581 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:44,581 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:44,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,582 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:44,583 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/7fcabd81360b4ef5995bc183ecdb1d2b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b 2023-04-13 02:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:44,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:44,607 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:44,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:44,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:44,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:44,607 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:44,609 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,609 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,609 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b, entries=250, sequenceid=265, filesize=16.6 K 2023-04-13 02:53:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,612 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:44,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:44,619 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:44,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/208ff3d80e494d5c8cac5b2612b97d7e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e 2023-04-13 02:53:44,619 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,619 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-13 02:53:44,620 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-13 02:53:44,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e, entries=150, sequenceid=265, filesize=11.9 K 2023-04-13 02:53:44,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/849d7f335c7c40cfa1c4b603635d64f1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1 2023-04-13 02:53:44,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:44,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354484638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1, entries=150, sequenceid=265, filesize=11.9 K 2023-04-13 02:53:44,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=138.65 KB/141980 for 23777bf909bb0929ddad8e9cb44ee33e in 828ms, sequenceid=265, compaction requested=true 2023-04-13 02:53:44,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:44,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354423988, max timestamp=1681354424659Num uniques 1068; 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354423988, max timestamp=1681354424659Num uniques 1068; 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354423988, max timestamp=1681354424659Num uniques 1018; 2023-04-13 02:53:44,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:53:44,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:44,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:44,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:44,668 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:44,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=33.54 KB heapSize=85.31 KB 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:44,672 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:44,684 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43374 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:44,684 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:44,687 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:44,687 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,687 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=32.9 K 2023-04-13 02:53:44,688 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:44,689 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,689 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=42.4 K 2023-04-13 02:53:44,693 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 89304eca7d6b4cd0af7c3f7ff30362dd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681354421416 2023-04-13 02:53:44,693 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d1c7b314755b497297f7e430f1286862, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681354421416 2023-04-13 02:53:44,712 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e77d80f620b9410e8e4e3d2cd08915b3, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681354422370 2023-04-13 02:53:44,719 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 7fcabd81360b4ef5995bc183ecdb1d2b, keycount=250, bloomtype=ROW, size=16.6 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:44,719 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:44,719 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting caa6b11ad3794397b56094efbac183ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681354422370 2023-04-13 02:53:44,724 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 208ff3d80e494d5c8cac5b2612b97d7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:44,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:44,772 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:44,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:44,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:44,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,791 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/103fa7f8c8eb40d5ba2c518ef09946ed] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:53:44,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/103fa7f8c8eb40d5ba2c518ef09946ed 2023-04-13 02:53:44,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:44,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:44,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:44,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:44,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:44,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:44,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:44,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe6592[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:44,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:44,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,813 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#10 average throughput is 0.36 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:44,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,818 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#11 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:44,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:44,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:44,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:44,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:44,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:44,824 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:44,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:44,831 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:44,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:44,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:44,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:44,831 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:44,834 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:44,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:44,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,840 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,845 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:44,846 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:44,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:44,846 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:44,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:44,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:44,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:44,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:44,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:44,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354424729, max timestamp=1681354424870Num uniques 859; 2023-04-13 02:53:44,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354424729, max timestamp=1681354424870Num uniques 859; 2023-04-13 02:53:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354424729, max timestamp=1681354424870Num uniques 859; 2023-04-13 02:53:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:44,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:44,879 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:44,979 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:44,990 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e90404b4187644e4a15f19fe68ca42eb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e90404b4187644e4a15f19fe68ca42eb 2023-04-13 02:53:44,991 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/53a3077d55854b11ac3d9d33af0cf85f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/53a3077d55854b11ac3d9d33af0cf85f 2023-04-13 02:53:44,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,010 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,010 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,015 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,015 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354424967, max timestamp=1681354425010Num uniques 1118; 2023-04-13 02:53:45,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354424967, max timestamp=1681354425010Num uniques 1118; 2023-04-13 02:53:45,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-13 02:53:45,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-13 02:53:45,022 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354424967, max timestamp=1681354425010Num uniques 1118; 2023-04-13 02:53:45,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-13 02:53:45,026 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 53a3077d55854b11ac3d9d33af0cf85f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:45,026 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:45,026 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=13, startTime=1681354424668; duration=0sec 2023-04-13 02:53:45,032 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:45,032 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:45,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,046 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into e90404b4187644e4a15f19fe68ca42eb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:45,047 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:45,047 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=13, startTime=1681354424663; duration=0sec 2023-04-13 02:53:45,047 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:45,047 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:45,048 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:45,048 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,048 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=32.9 K 2023-04-13 02:53:45,049 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,050 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,050 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,051 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8ecdc06b82404d80b78712fff33f1a0f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681354421416 2023-04-13 02:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,062 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6f29182a1bd443e88196dc4b6b25f617, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681354422370 2023-04-13 02:53:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,063 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 849d7f335c7c40cfa1c4b603635d64f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71fdc2f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,073 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,073 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,077 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:45,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,086 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,086 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,109 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#13 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681354425109, max timestamp=1681354425135Num uniques 1027; 2023-04-13 02:53:45,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-13 02:53:45,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681354425109, max timestamp=1681354425135Num uniques 1027; 2023-04-13 02:53:45,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-13 02:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681354425109, max timestamp=1681354425135Num uniques 1027; 2023-04-13 02:53:45,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-13 02:53:45,176 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/23502074dbae4bb6a7225ecc9e03e9c9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23502074dbae4bb6a7225ecc9e03e9c9 2023-04-13 02:53:45,186 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:45,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:45,188 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 23502074dbae4bb6a7225ecc9e03e9c9(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:45,188 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:45,188 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=13, startTime=1681354424668; duration=0sec 2023-04-13 02:53:45,189 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:45,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,199 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,208 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,208 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354485197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354485200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,210 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354485206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,219 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354485209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354485207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471ac7a5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,227 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,229 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,239 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,239 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,250 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,257 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,257 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,260 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,267 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,267 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,267 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354485316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354485317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354485321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354485322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354485323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:45,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,375 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,379 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,379 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/d74eb3da9d13472dae45afadcab7ece7 2023-04-13 02:53:45,392 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,392 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,392 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,393 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,395 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,395 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,395 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef7a3c1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,402 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,406 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,412 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,413 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/f6fc36966c344d808674c3473b99db6c 2023-04-13 02:53:45,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,445 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,445 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:45,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/103fa7f8c8eb40d5ba2c518ef09946ed as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed 2023-04-13 02:53:45,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed, entries=250, sequenceid=304, filesize=16.7 K 2023-04-13 02:53:45,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/d74eb3da9d13472dae45afadcab7ece7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7 2023-04-13 02:53:45,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7, entries=150, sequenceid=304, filesize=11.9 K 2023-04-13 02:53:45,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/f6fc36966c344d808674c3473b99db6c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c 2023-04-13 02:53:45,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c, entries=150, sequenceid=304, filesize=11.9 K 2023-04-13 02:53:45,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=154.31 KB/158010 for 23777bf909bb0929ddad8e9cb44ee33e in 834ms, sequenceid=304, compaction requested=false 2023-04-13 02:53:45,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:45,513 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 02:53:45,513 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-13 02:53:45,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354425199, max timestamp=1681354425522Num uniques 1186; 2023-04-13 02:53:45,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-13 02:53:45,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=116.29 KB heapSize=320.98 KB 2023-04-13 02:53:45,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354425199, max timestamp=1681354425524Num uniques 1186; 2023-04-13 02:53:45,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-13 02:53:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:45,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354425199, max timestamp=1681354425522Num uniques 1186; 2023-04-13 02:53:45,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-13 02:53:45,530 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:45,546 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:45,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:45,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/96dc41ca5f93447d9af9ae57e85a659b 2023-04-13 02:53:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,575 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,575 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,596 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,596 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354425552, max timestamp=1681354425597Num uniques 859; 2023-04-13 02:53:45,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:45,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354425552, max timestamp=1681354425598Num uniques 859; 2023-04-13 02:53:45,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354425552, max timestamp=1681354425598Num uniques 859; 2023-04-13 02:53:45,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:45,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d269b78[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,612 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,612 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,615 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:45,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,622 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,622 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,667 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,667 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,667 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,667 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,667 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,667 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354425626, max timestamp=1681354425693Num uniques 1018; 2023-04-13 02:53:45,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:53:45,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354425626, max timestamp=1681354425693Num uniques 1018; 2023-04-13 02:53:45,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:53:45,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354425626, max timestamp=1681354425693Num uniques 1018; 2023-04-13 02:53:45,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:53:45,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681354425705, max timestamp=1681354425725Num uniques 1027; 2023-04-13 02:53:45,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-13 02:53:45,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681354425705, max timestamp=1681354425725Num uniques 1027; 2023-04-13 02:53:45,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-13 02:53:45,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681354425705, max timestamp=1681354425725Num uniques 1027; 2023-04-13 02:53:45,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-13 02:53:45,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354485748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354485751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354485753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354485755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354485756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,767 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:45,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,776 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,783 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,783 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,786 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1de57[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,801 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,801 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,811 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,812 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354485857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354485859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354485861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354485862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354485862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:45,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:45,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:45,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:45,924 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:45,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:45,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:45,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:45,946 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:45,946 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:45,958 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:45,958 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:45,958 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,958 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:45,958 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:45,959 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:45,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:45,961 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,961 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:45,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15170689[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:45,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:45,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:45,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:45,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:45,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:45,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:45,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:45,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:45,975 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:45,975 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:45,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:45,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:45,976 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:45,980 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,980 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,980 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:45,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:45,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,983 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:45,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:45,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:45,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:45,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:45,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:45,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:45,987 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:45,988 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:45,989 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:45,989 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:45,989 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,989 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:45,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:45,989 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354486059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354486066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354486068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354486069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354486069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/8af79007fa604b5eacf1ad7e60e50be0 2023-04-13 02:53:46,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:46,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:46,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:46,104 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:46,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:46,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:46,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,109 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:46,127 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:46,130 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,130 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:46,130 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:46,130 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:46,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,133 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:46,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd4641[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:46,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:46,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,141 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:46,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:46,145 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:46,151 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:46,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:46,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:46,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:46,151 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:46,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:46,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:46,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:46,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:46,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:46,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:46,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:46,278 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:46,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:46,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:46,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,285 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,285 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:46,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:46,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,288 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:46,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:46,289 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,289 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:46,296 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:46,296 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:46,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,298 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f89eab5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:46,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:46,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:46,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:46,307 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:46,313 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:46,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:46,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:46,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:46,313 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:46,315 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,324 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:46,327 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:46,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:46,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354486365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52076 deadline: 1681354486375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354486376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354486376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52072 deadline: 1681354486381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:46,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:46,436 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:46,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:46,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:46,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,439 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,440 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,440 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:46,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:46,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:46,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:46,443 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:46,443 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,443 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:46,444 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,444 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:46,444 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:46,444 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:46,446 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1550cc75[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:46,448 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:46,448 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,449 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:46,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:46,452 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:46,456 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:46,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:46,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:46,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:46,456 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:46,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:46,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,483 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,488 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,488 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:46,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,488 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:46,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,489 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:46,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:46,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:46,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:46,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:46,489 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:46,489 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:46,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/1da447461457452487448bb4dba5032c 2023-04-13 02:53:46,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/96dc41ca5f93447d9af9ae57e85a659b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b 2023-04-13 02:53:46,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b, entries=250, sequenceid=380, filesize=16.7 K 2023-04-13 02:53:46,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/8af79007fa604b5eacf1ad7e60e50be0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0 2023-04-13 02:53:46,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0, entries=150, sequenceid=380, filesize=11.9 K 2023-04-13 02:53:46,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/1da447461457452487448bb4dba5032c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c 2023-04-13 02:53:46,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c, entries=150, sequenceid=380, filesize=11.9 K 2023-04-13 02:53:46,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~83.58 KB/85584, currentSize=156.54 KB/160300 for 23777bf909bb0929ddad8e9cb44ee33e in 1055ms, sequenceid=380, compaction requested=true 2023-04-13 02:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:46,585 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:46,585 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:46,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:46,589 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36611 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:46,589 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46371 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:46,589 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:46,590 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:46,590 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:46,590 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,590 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,590 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e90404b4187644e4a15f19fe68ca42eb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=45.3 K 2023-04-13 02:53:46,590 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/53a3077d55854b11ac3d9d33af0cf85f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=35.8 K 2023-04-13 02:53:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:46,592 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 53a3077d55854b11ac3d9d33af0cf85f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:46,592 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e90404b4187644e4a15f19fe68ca42eb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:46,593 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d74eb3da9d13472dae45afadcab7ece7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681354423991 2023-04-13 02:53:46,594 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8af79007fa604b5eacf1ad7e60e50be0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425207 2023-04-13 02:53:46,595 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 103fa7f8c8eb40d5ba2c518ef09946ed, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681354423988 2023-04-13 02:53:46,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:46,599 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 96dc41ca5f93447d9af9ae57e85a659b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425199 2023-04-13 02:53:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:46,604 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:46,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:46,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:46,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:46,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:46,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:46,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:46,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:46,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:46,628 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:46,628 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=156.54 KB heapSize=434.08 KB 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:46,629 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:46,630 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#18 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:46,653 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#19 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:46,736 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/7b47be0c018540918dcd7bc49140c400] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:53:46,740 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/2d0aa2a71b394232b290f6095c159804 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2d0aa2a71b394232b290f6095c159804 2023-04-13 02:53:46,747 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/7b47be0c018540918dcd7bc49140c400 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/7b47be0c018540918dcd7bc49140c400 2023-04-13 02:53:46,754 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into 2d0aa2a71b394232b290f6095c159804(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:46,754 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:46,754 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=13, startTime=1681354426585; duration=0sec 2023-04-13 02:53:46,754 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:46,754 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:46,761 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36611 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:46,761 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:46,761 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:46,762 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23502074dbae4bb6a7225ecc9e03e9c9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=35.8 K 2023-04-13 02:53:46,762 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 23502074dbae4bb6a7225ecc9e03e9c9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681354422529 2023-04-13 02:53:46,763 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f6fc36966c344d808674c3473b99db6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681354423991 2023-04-13 02:53:46,764 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 1da447461457452487448bb4dba5032c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425207 2023-04-13 02:53:46,769 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 7b47be0c018540918dcd7bc49140c400(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:46,772 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:46,772 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=13, startTime=1681354426585; duration=0sec 2023-04-13 02:53:46,772 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:46,788 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:46,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/23e6753790a74209a506f8ee44d53650 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23e6753790a74209a506f8ee44d53650 2023-04-13 02:53:46,836 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 23e6753790a74209a506f8ee44d53650(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:46,836 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:46,836 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=13, startTime=1681354426585; duration=0sec 2023-04-13 02:53:46,836 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:46,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354486885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354486888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354486907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354486907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:46,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:46,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354486907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354487016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354487017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354487017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,129 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/254a2f2151ba4064bd2532cba37da7fb 2023-04-13 02:53:47,166 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/1eb1ebe7a74943d5b489542a0d3cae66 2023-04-13 02:53:47,214 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/79086c5ff2f64faea00c797353d8ecb7 2023-04-13 02:53:47,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52106 deadline: 1681354487220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354487220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354487220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,227 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/254a2f2151ba4064bd2532cba37da7fb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb 2023-04-13 02:53:47,237 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb, entries=150, sequenceid=451, filesize=11.9 K 2023-04-13 02:53:47,240 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/1eb1ebe7a74943d5b489542a0d3cae66 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66 2023-04-13 02:53:47,250 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66, entries=150, sequenceid=451, filesize=11.9 K 2023-04-13 02:53:47,251 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/79086c5ff2f64faea00c797353d8ecb7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7 2023-04-13 02:53:47,264 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7, entries=150, sequenceid=451, filesize=11.9 K 2023-04-13 02:53:47,268 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~156.54 KB/160300, heapSize ~434.03 KB/444448, currentSize=26.84 KB/27480 for 23777bf909bb0929ddad8e9cb44ee33e in 640ms, sequenceid=451, compaction requested=false 2023-04-13 02:53:47,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:47,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:47,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:47,269 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:47,273 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38dea318[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:47,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:47,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,279 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:47,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:47,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:47,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:47,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:47,283 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:47,286 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:47,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:47,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:47,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:47,287 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:47,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:47,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,300 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:47,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,305 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:47,305 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:47,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:47,405 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:47,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:47,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:47,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:47,410 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:47,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:47,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:47,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:47,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,412 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:47,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:47,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:47,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:47,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:47,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:47,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:47,415 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:53:47,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:47,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:47,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:47,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:47,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:47,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:47,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:47,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:47,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354427545, max timestamp=1681354427574Num uniques 859; 2023-04-13 02:53:47,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354427545, max timestamp=1681354427574Num uniques 859; 2023-04-13 02:53:47,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:47,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354427545, max timestamp=1681354427574Num uniques 859; 2023-04-13 02:53:47,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:47,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:47,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354427588, max timestamp=1681354427632Num uniques 1068; 2023-04-13 02:53:47,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 02:53:47,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354427588, max timestamp=1681354427632Num uniques 1068; 2023-04-13 02:53:47,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 02:53:47,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354427588, max timestamp=1681354427632Num uniques 1068; 2023-04-13 02:53:47,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681354427648, max timestamp=1681354427659Num uniques 1127; 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681354427648, max timestamp=1681354427659Num uniques 1127; 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681354427648, max timestamp=1681354427659Num uniques 1127; 2023-04-13 02:53:47,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-13 02:53:47,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354487686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354487688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354487689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354487791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52070 deadline: 1681354487794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354487794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,838 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/aeac256c6a34458e94b8e116385c5dd4 2023-04-13 02:53:47,877 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/d305af5b5b424739a41fb2876cf4282c 2023-04-13 02:53:47,917 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/b70ebde0db39466c86cb338104cfd4f0 2023-04-13 02:53:47,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354487916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:47,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354487922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,930 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/aeac256c6a34458e94b8e116385c5dd4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4 2023-04-13 02:53:47,938 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4, entries=100, sequenceid=461, filesize=9.5 K 2023-04-13 02:53:47,940 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/d305af5b5b424739a41fb2876cf4282c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c 2023-04-13 02:53:47,949 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c, entries=100, sequenceid=461, filesize=9.5 K 2023-04-13 02:53:47,951 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/b70ebde0db39466c86cb338104cfd4f0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0 2023-04-13 02:53:47,957 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0, entries=100, sequenceid=461, filesize=9.5 K 2023-04-13 02:53:47,959 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 23777bf909bb0929ddad8e9cb44ee33e in 544ms, sequenceid=461, compaction requested=true 2023-04-13 02:53:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:47,959 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:47,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:47,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed750c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:47,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:47,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,965 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:47,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,966 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:47,967 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:47,970 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:47,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:47,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:47,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:47,970 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:47,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:47,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:47,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:47,975 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,978 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:47,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:47,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:47,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:47,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:47,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:47,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:47,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:47,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:47,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:47,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354427684, max timestamp=1681354427998Num uniques 1236; 2023-04-13 02:53:48,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-13 02:53:48,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=127.47 KB heapSize=351.06 KB 2023-04-13 02:53:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:48,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354427684, max timestamp=1681354428000Num uniques 1236; 2023-04-13 02:53:48,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=8516, suffix heap size=156512,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=186 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:48,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:53:48,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e68d6d84a46448b6a966c70918f341be 2023-04-13 02:53:48,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354488049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354488050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354488053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,082 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,093 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f9e93af853f84001bfdb7b5e5cefe092 2023-04-13 02:53:48,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,095 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,096 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,096 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,122 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,122 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,122 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,122 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,129 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e1eb66[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,135 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,135 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,138 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,138 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,138 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,138 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,141 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,142 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,144 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:48,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354488155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354488160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354488160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,260 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,260 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,260 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,260 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,260 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,260 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,262 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:48,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,265 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,265 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,265 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,265 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,267 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4741a6eb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,269 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,269 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,272 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,272 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,272 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,272 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,275 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,275 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,283 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,283 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,283 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,283 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,283 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:48,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354488365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354488368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354488369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,383 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,397 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,399 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,400 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:48,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,408 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,409 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,409 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,409 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,412 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,412 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,412 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aca0b78[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,415 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,421 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,422 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,435 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,435 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,535 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,546 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,553 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/644296831c5a4201a766f8e2fb42c741 2023-04-13 02:53:48,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e68d6d84a46448b6a966c70918f341be as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be 2023-04-13 02:53:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,596 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,596 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,596 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,596 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be, entries=300, sequenceid=535, filesize=19.0 K 2023-04-13 02:53:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,601 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f9e93af853f84001bfdb7b5e5cefe092 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092 2023-04-13 02:53:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e05a38[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092, entries=150, sequenceid=535, filesize=11.9 K 2023-04-13 02:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/644296831c5a4201a766f8e2fb42c741 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741 2023-04-13 02:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,622 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,622 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,624 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,624 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,624 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,625 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,630 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741, entries=150, sequenceid=535, filesize=11.9 K 2023-04-13 02:53:48,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,631 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~237.22 KB/242912, currentSize=93.93 KB/96180 for 23777bf909bb0929ddad8e9cb44ee33e in 629ms, sequenceid=535, compaction requested=true 2023-04-13 02:53:48,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:48,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:48,633 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:48,633 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:48,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:48,635 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53826 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:48,635 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:48,635 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,635 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2d0aa2a71b394232b290f6095c159804, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=52.6 K 2023-04-13 02:53:48,636 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:48,636 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:48,636 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,636 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/7b47be0c018540918dcd7bc49140c400, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=45.4 K 2023-04-13 02:53:48,636 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2d0aa2a71b394232b290f6095c159804, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425207 2023-04-13 02:53:48,638 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7b47be0c018540918dcd7bc49140c400, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425207 2023-04-13 02:53:48,639 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 254a2f2151ba4064bd2532cba37da7fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681354425751 2023-04-13 02:53:48,640 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1eb1ebe7a74943d5b489542a0d3cae66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681354425751 2023-04-13 02:53:48,640 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting aeac256c6a34458e94b8e116385c5dd4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681354426878 2023-04-13 02:53:48,641 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d305af5b5b424739a41fb2876cf4282c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681354426878 2023-04-13 02:53:48,642 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e68d6d84a46448b6a966c70918f341be, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427684 2023-04-13 02:53:48,642 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f9e93af853f84001bfdb7b5e5cefe092, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427687 2023-04-13 02:53:48,670 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:48,678 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:48,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:48,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:53:48,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:48,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:48,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:48,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:48,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:48,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:48,732 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354488728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,740 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,740 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,740 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,740 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,742 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,742 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354488740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,743 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,743 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,744 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354488740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,744 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,752 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,752 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,752 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,752 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,757 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e813739[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,763 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,777 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,778 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:48,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,791 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354488841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354488845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:48,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354488845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:48,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:48,895 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:48,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:48,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:48,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:48,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:48,899 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,900 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:48,900 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:48,900 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:48,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74bda8fc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:48,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:48,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:48,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:48,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:48,908 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:48,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:48,912 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:48,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:48,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:48,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:48,912 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:48,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:48,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,914 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:48,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:48,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:48,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:48,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:48,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:48,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:48,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:48,920 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:48,920 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:48,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:48,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:49,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:49,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:49,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:49,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:49,033 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:49,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:49,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:49,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:49,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:49,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:49,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:49,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:49,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:49,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:49,040 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e76ecd8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:49,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:49,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:49,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354489044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:49,051 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:49,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:49,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,054 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:49,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:49,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:49,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:49,054 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:49,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354489051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354489052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:49,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:49,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:49,069 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:49,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,117 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/4937f22d36ea40ca9d38efb2d1293841 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4937f22d36ea40ca9d38efb2d1293841 2023-04-13 02:53:49,119 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/0104fa72daa649cdaa6d92ceff35479f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/0104fa72daa649cdaa6d92ceff35479f 2023-04-13 02:53:49,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/b3d688fd21404922b08f56bd477d4645 2023-04-13 02:53:49,133 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into 4937f22d36ea40ca9d38efb2d1293841(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,133 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,134 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=12, startTime=1681354428632; duration=0sec 2023-04-13 02:53:49,134 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:49,134 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:49,137 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46506 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:49,138 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:49,138 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,138 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23e6753790a74209a506f8ee44d53650, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=45.4 K 2023-04-13 02:53:49,138 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 23e6753790a74209a506f8ee44d53650, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681354425207 2023-04-13 02:53:49,139 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 79086c5ff2f64faea00c797353d8ecb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681354425751 2023-04-13 02:53:49,139 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 0104fa72daa649cdaa6d92ceff35479f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,140 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,140 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=12, startTime=1681354428633; duration=0sec 2023-04-13 02:53:49,141 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,142 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b70ebde0db39466c86cb338104cfd4f0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681354426878 2023-04-13 02:53:49,142 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 644296831c5a4201a766f8e2fb42c741, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427687 2023-04-13 02:53:49,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/3d8b1b74ca3545439a9d289caa21d20a 2023-04-13 02:53:49,162 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:49,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:49,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:49,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:49,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:49,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:49,175 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:49,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:49,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:49,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,177 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:49,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:49,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:49,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:49,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:49,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:49,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,190 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:49,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/cfa3e68d41b641e8aff99eba2b511bd4 2023-04-13 02:53:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492052b4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:49,194 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:49,194 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,198 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:49,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:49,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/b3d688fd21404922b08f56bd477d4645 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645 2023-04-13 02:53:49,200 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:49,204 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:49,204 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:49,204 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:49,204 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:49,204 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:49,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645, entries=200, sequenceid=554, filesize=14.3 K 2023-04-13 02:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/3d8b1b74ca3545439a9d289caa21d20a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a 2023-04-13 02:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:49,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:49,218 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:49,218 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:49,219 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:49,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a, entries=150, sequenceid=554, filesize=11.9 K 2023-04-13 02:53:49,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/cfa3e68d41b641e8aff99eba2b511bd4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4 2023-04-13 02:53:49,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4, entries=150, sequenceid=554, filesize=11.9 K 2023-04-13 02:53:49,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 23777bf909bb0929ddad8e9cb44ee33e in 558ms, sequenceid=554, compaction requested=false 2023-04-13 02:53:49,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,319 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:49,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:49,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:49,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:49,326 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:49,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:49,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:49,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:49,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:49,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:49,332 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,332 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:49,332 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:53:49,332 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:49,332 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,332 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:49,333 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,333 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:49,333 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,348 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/12c838d01e7b48b2b0d1ae6695e8bd18 2023-04-13 02:53:49,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:49,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:49,376 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f26af0891d4e4d8586de55ae71a11186 2023-04-13 02:53:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354489379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354489381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354489382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,410 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/c74a674d59c6460f8a9728b7d47adbbb 2023-04-13 02:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/12c838d01e7b48b2b0d1ae6695e8bd18 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18 2023-04-13 02:53:49,428 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18, entries=150, sequenceid=570, filesize=11.9 K 2023-04-13 02:53:49,429 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f26af0891d4e4d8586de55ae71a11186 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186 2023-04-13 02:53:49,443 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186, entries=150, sequenceid=570, filesize=11.9 K 2023-04-13 02:53:49,447 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/c74a674d59c6460f8a9728b7d47adbbb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb 2023-04-13 02:53:49,482 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb, entries=150, sequenceid=570, filesize=11.9 K 2023-04-13 02:53:49,483 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 23777bf909bb0929ddad8e9cb44ee33e in 151ms, sequenceid=570, compaction requested=true 2023-04-13 02:53:49,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,488 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 02:53:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eaf8c56[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:49,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:49,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,496 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:49,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:49,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:49,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:49,501 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:49,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:49,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:49,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:49,501 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:49,503 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,503 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,503 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:49,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,513 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:49,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:49,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,513 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:49,513 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:49,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354489522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354489527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354489529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,552 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/9e803e3c971b461994b78412c4acb36b 2023-04-13 02:53:49,586 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/fbde30cf9e744bdaa4040c3b33e7a5c1] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:53:49,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/fbde30cf9e744bdaa4040c3b33e7a5c1 2023-04-13 02:53:49,593 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/edbb690fa77c47088383d00f8a03e30f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/edbb690fa77c47088383d00f8a03e30f 2023-04-13 02:53:49,604 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into edbb690fa77c47088383d00f8a03e30f(size=12.2 K), total size for store is 36.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,604 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,604 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=12, startTime=1681354428633; duration=0sec 2023-04-13 02:53:49,604 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/bdc14121925142d6a7ef745071db5552 2023-04-13 02:53:49,613 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:49,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:49,623 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:49,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:49,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:49,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/9e803e3c971b461994b78412c4acb36b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b 2023-04-13 02:53:49,624 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,625 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,626 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:49,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b, entries=200, sequenceid=590, filesize=14.3 K 2023-04-13 02:53:49,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/fbde30cf9e744bdaa4040c3b33e7a5c1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1 2023-04-13 02:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:49,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:49,639 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,639 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:49,639 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:49,639 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:49,642 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354489639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1, entries=150, sequenceid=590, filesize=11.9 K 2023-04-13 02:53:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354489640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354489641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/bdc14121925142d6a7ef745071db5552 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552 2023-04-13 02:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674c3137[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:49,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:49,653 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:49,657 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:49,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:49,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:49,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:49,657 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:49,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552, entries=150, sequenceid=590, filesize=11.9 K 2023-04-13 02:53:49,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 23777bf909bb0929ddad8e9cb44ee33e in 169ms, sequenceid=590, compaction requested=true 2023-04-13 02:53:49,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,662 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:49,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,663 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:49,665 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53966 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:49,665 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:49,665 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,665 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4937f22d36ea40ca9d38efb2d1293841, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=52.7 K 2023-04-13 02:53:49,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,668 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:49,667 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4937f22d36ea40ca9d38efb2d1293841, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427687 2023-04-13 02:53:49,669 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:49,669 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,669 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/0104fa72daa649cdaa6d92ceff35479f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=47.9 K 2023-04-13 02:53:49,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:49,674 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0104fa72daa649cdaa6d92ceff35479f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427687 2023-04-13 02:53:49,673 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b3d688fd21404922b08f56bd477d4645, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1681354428043 2023-04-13 02:53:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:49,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:49,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:49,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:49,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:49,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:49,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:49,682 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3d8b1b74ca3545439a9d289caa21d20a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1681354428043 2023-04-13 02:53:49,682 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 12c838d01e7b48b2b0d1ae6695e8bd18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681354428688 2023-04-13 02:53:49,683 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f26af0891d4e4d8586de55ae71a11186, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681354428688 2023-04-13 02:53:49,683 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9e803e3c971b461994b78412c4acb36b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429377 2023-04-13 02:53:49,684 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting fbde30cf9e744bdaa4040c3b33e7a5c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429379 2023-04-13 02:53:49,704 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:49,706 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:49,772 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/04ff373b0b144c38803928620eae059e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/04ff373b0b144c38803928620eae059e 2023-04-13 02:53:49,779 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/ce6ea08fe5e9447aa5aa526a86fae079 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/ce6ea08fe5e9447aa5aa526a86fae079 2023-04-13 02:53:49,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:49,782 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into 04ff373b0b144c38803928620eae059e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,782 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=12, startTime=1681354429662; duration=0sec 2023-04-13 02:53:49,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:49,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:49,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:53:49,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:49,786 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:53:49,786 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:49,786 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,787 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/edbb690fa77c47088383d00f8a03e30f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=47.9 K 2023-04-13 02:53:49,787 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting edbb690fa77c47088383d00f8a03e30f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681354427687 2023-04-13 02:53:49,788 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting cfa3e68d41b641e8aff99eba2b511bd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1681354428043 2023-04-13 02:53:49,788 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c74a674d59c6460f8a9728b7d47adbbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681354428688 2023-04-13 02:53:49,790 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bdc14121925142d6a7ef745071db5552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429379 2023-04-13 02:53:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:49,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:49,804 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:49,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:49,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:49,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,805 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into ce6ea08fe5e9447aa5aa526a86fae079(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:49,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:49,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:49,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:49,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:49,805 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,809 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=12, startTime=1681354429663; duration=0sec 2023-04-13 02:53:49,810 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:49,820 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:49,821 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,821 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:49,821 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,821 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:49,821 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:49,829 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:49,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:49,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:49,854 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d4c39c0c392343068d9214f7df0d449e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d4c39c0c392343068d9214f7df0d449e 2023-04-13 02:53:49,861 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/db54ed5554e044d18fc5b99a14e47ba3 2023-04-13 02:53:49,862 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into d4c39c0c392343068d9214f7df0d449e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:49,862 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:49,862 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=12, startTime=1681354429663; duration=0sec 2023-04-13 02:53:49,863 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:49,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354489894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354489896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354489895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354489929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,934 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:53:49,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354489939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:49,945 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:53:50,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354490002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354490002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354490002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354490208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354490210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354490211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,329 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/09d5d51ccdbe451aaafa3c709e9632e2 2023-04-13 02:53:50,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354490512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354490514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354490515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,753 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/67880cb78e3d402295e04d8e5e1f653f 2023-04-13 02:53:50,761 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/db54ed5554e044d18fc5b99a14e47ba3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3 2023-04-13 02:53:50,768 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3, entries=150, sequenceid=605, filesize=11.9 K 2023-04-13 02:53:50,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/09d5d51ccdbe451aaafa3c709e9632e2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2 2023-04-13 02:53:50,776 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2, entries=150, sequenceid=605, filesize=11.9 K 2023-04-13 02:53:50,777 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/67880cb78e3d402295e04d8e5e1f653f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f 2023-04-13 02:53:50,795 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f, entries=150, sequenceid=605, filesize=11.9 K 2023-04-13 02:53:50,796 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 23777bf909bb0929ddad8e9cb44ee33e in 976ms, sequenceid=605, compaction requested=false 2023-04-13 02:53:50,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:50,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:50,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e45012a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:50,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:50,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:50,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:50,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:50,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:50,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:50,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:50,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:50,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:50,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:50,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:50,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:50,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:50,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:50,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:50,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,821 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:50,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:50,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:50,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:50,826 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:50,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:50,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:50,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:50,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:50,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:50,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:50,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:50,932 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:50,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:50,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:50,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:50,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:50,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:50,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:50,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:50,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:50,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:50,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:50,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:50,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:50,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:50,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:50,937 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:50,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:51,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,353 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/584ad7689df74710841beb49324e1ebe 2023-04-13 02:53:51,390 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/3ca013d92947430294012cd70246054d 2023-04-13 02:53:51,406 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=626 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/c92bc618f51f44aa8c432091dfacc94b 2023-04-13 02:53:51,413 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/584ad7689df74710841beb49324e1ebe as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe 2023-04-13 02:53:51,419 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe, entries=150, sequenceid=626, filesize=11.9 K 2023-04-13 02:53:51,420 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/3ca013d92947430294012cd70246054d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d 2023-04-13 02:53:51,427 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d, entries=150, sequenceid=626, filesize=11.9 K 2023-04-13 02:53:51,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/c92bc618f51f44aa8c432091dfacc94b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b 2023-04-13 02:53:51,434 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b, entries=150, sequenceid=626, filesize=11.9 K 2023-04-13 02:53:51,435 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 23777bf909bb0929ddad8e9cb44ee33e in 498ms, sequenceid=626, compaction requested=true 2023-04-13 02:53:51,435 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:51,435 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:51,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353b49a4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:51,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:51,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:51,445 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,445 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:51,447 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:51,447 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:51,450 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:51,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:51,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:51,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:51,450 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:51,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:51,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:51,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:51,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:51,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:51,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:51,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:51,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:51,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:51,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:51,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:51,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:51,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:51,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:51,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:51,581 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:51,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:51,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:51,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:51,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:51,583 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:51,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:51,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:51,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:51,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:51,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:51,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:51,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:51,585 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:53:51,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:51,586 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:51,586 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:51,586 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:51,586 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:51,586 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:51,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:51,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:51,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354491977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354491977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:51,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354491980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,003 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/d5eef4bd56ae40ee9998a7667a9fa751 2023-04-13 02:53:52,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354492281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354492281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354492284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,424 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/afa96c83d0274723b92e2322c316aa83 2023-04-13 02:53:52,442 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/919e59cfd69f46a09b072bb766394730 2023-04-13 02:53:52,449 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/d5eef4bd56ae40ee9998a7667a9fa751 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751 2023-04-13 02:53:52,456 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751, entries=150, sequenceid=638, filesize=11.9 K 2023-04-13 02:53:52,457 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/afa96c83d0274723b92e2322c316aa83 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83 2023-04-13 02:53:52,464 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83, entries=150, sequenceid=638, filesize=11.9 K 2023-04-13 02:53:52,465 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/919e59cfd69f46a09b072bb766394730 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730 2023-04-13 02:53:52,476 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730, entries=150, sequenceid=638, filesize=11.9 K 2023-04-13 02:53:52,477 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 23777bf909bb0929ddad8e9cb44ee33e in 892ms, sequenceid=638, compaction requested=true 2023-04-13 02:53:52,477 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:52,477 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:52,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:52,486 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f4ae69[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:52,489 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:52,489 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,490 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:52,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:52,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:52,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:52,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:52,491 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:52,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:52,496 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:52,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:52,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:52,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:52,496 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:52,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,501 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:52,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:52,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:52,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:52,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:52,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:52,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:52,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:52,507 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,507 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,507 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:52,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:52,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:52,507 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:52,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:52,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:52,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:52,629 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:52,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:52,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:52,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:52,632 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,632 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:52,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,634 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:52,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:52,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:52,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:52,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:52,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:52,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:52,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:52,657 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:52,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:52,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:52,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:52,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354492802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354492804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354492806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354492907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354492907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:52,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:52,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354492913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,101 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/0c10a8e9bbe3472eb08f7da56996d681 2023-04-13 02:53:53,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354493111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354493113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354493118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,121 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/a7b01bb70e674be4917f3bcc5237c019 2023-04-13 02:53:53,147 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d9fcccecbea640ca8223fbbb183f75c8 2023-04-13 02:53:53,158 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/0c10a8e9bbe3472eb08f7da56996d681 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681 2023-04-13 02:53:53,165 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681, entries=150, sequenceid=658, filesize=11.9 K 2023-04-13 02:53:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/a7b01bb70e674be4917f3bcc5237c019 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019 2023-04-13 02:53:53,179 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019, entries=150, sequenceid=658, filesize=11.9 K 2023-04-13 02:53:53,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d9fcccecbea640ca8223fbbb183f75c8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8 2023-04-13 02:53:53,192 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8, entries=150, sequenceid=658, filesize=11.9 K 2023-04-13 02:53:53,193 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 23777bf909bb0929ddad8e9cb44ee33e in 536ms, sequenceid=658, compaction requested=true 2023-04-13 02:53:53,194 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,194 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:53,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:53,198 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,198 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3137f429[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:53,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:53,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:53,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:53,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:53,216 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:53,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:53,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:53,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:53,217 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:53,222 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,222 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,222 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:53,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:53,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:53,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,235 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,234 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:53,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:53,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:53,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:53,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:53,346 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:53,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:53,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:53,347 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,347 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,349 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:53,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:53,357 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,357 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:53,360 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,360 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:53,361 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:53,361 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,379 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/f6a62df11c3749358523995b9a58a865 2023-04-13 02:53:53,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:53,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:53,431 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/516e42b78e7f4120a889939bebc41066 2023-04-13 02:53:53,450 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/e587e45406b64f548dde6e5a5b2db465 2023-04-13 02:53:53,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354493453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354433443, max timestamp=1681354433454Num uniques 859; 2023-04-13 02:53:53,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354433443, max timestamp=1681354433454Num uniques 859; 2023-04-13 02:53:53,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:53,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354433443, max timestamp=1681354433454Num uniques 859; 2023-04-13 02:53:53,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:53,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:53:53,465 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/f6a62df11c3749358523995b9a58a865 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865 2023-04-13 02:53:53,472 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865, entries=150, sequenceid=670, filesize=11.9 K 2023-04-13 02:53:53,475 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/516e42b78e7f4120a889939bebc41066 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066 2023-04-13 02:53:53,486 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066, entries=150, sequenceid=670, filesize=11.9 K 2023-04-13 02:53:53,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/e587e45406b64f548dde6e5a5b2db465 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465 2023-04-13 02:53:53,493 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465, entries=150, sequenceid=670, filesize=11.9 K 2023-04-13 02:53:53,494 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=93.93 KB/96180 for 23777bf909bb0929ddad8e9cb44ee33e in 133ms, sequenceid=670, compaction requested=true 2023-04-13 02:53:53,494 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,494 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:53,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=100.63 KB heapSize=285.38 KB 2023-04-13 02:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:53,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672dff74[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:53,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:53,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:53,517 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,517 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:53,518 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:53,530 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:53,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:53,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:53,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:53,530 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:53,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:53,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,539 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:53,539 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/59828948cccd4bf2a4e5e1e81aee4765 2023-04-13 02:53:53,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354493543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354493544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354493558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/607bce681fb944e9a17964874de6fe91 2023-04-13 02:53:53,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/9182e0c0d64b44d782b843aea480868f 2023-04-13 02:53:53,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/59828948cccd4bf2a4e5e1e81aee4765 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765 2023-04-13 02:53:53,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765, entries=350, sequenceid=703, filesize=21.4 K 2023-04-13 02:53:53,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/607bce681fb944e9a17964874de6fe91 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91 2023-04-13 02:53:53,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91, entries=150, sequenceid=703, filesize=11.9 K 2023-04-13 02:53:53,608 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-13 02:53:53,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/9182e0c0d64b44d782b843aea480868f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f 2023-04-13 02:53:53,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f, entries=150, sequenceid=703, filesize=11.9 K 2023-04-13 02:53:53,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=73.80 KB/75570 for 23777bf909bb0929ddad8e9cb44ee33e in 103ms, sequenceid=703, compaction requested=true 2023-04-13 02:53:53,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:53,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:53,615 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:53:53,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:53:53,615 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:53:53,621 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 95544 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:53:53,621 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:53,621 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:53:53,621 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,622 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:53,622 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/04ff373b0b144c38803928620eae059e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=93.3 K 2023-04-13 02:53:53,622 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,622 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/ce6ea08fe5e9447aa5aa526a86fae079, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=83.8 K 2023-04-13 02:53:53,622 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 04ff373b0b144c38803928620eae059e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429379 2023-04-13 02:53:53,622 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ce6ea08fe5e9447aa5aa526a86fae079, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429379 2023-04-13 02:53:53,623 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting db54ed5554e044d18fc5b99a14e47ba3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354429505 2023-04-13 02:53:53,623 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 09d5d51ccdbe451aaafa3c709e9632e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354429505 2023-04-13 02:53:53,623 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 584ad7689df74710841beb49324e1ebe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681354429887 2023-04-13 02:53:53,623 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3ca013d92947430294012cd70246054d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681354429887 2023-04-13 02:53:53,623 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d5eef4bd56ae40ee9998a7667a9fa751, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681354431026 2023-04-13 02:53:53,625 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting afa96c83d0274723b92e2322c316aa83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681354431026 2023-04-13 02:53:53,625 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0c10a8e9bbe3472eb08f7da56996d681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681354431653 2023-04-13 02:53:53,626 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a7b01bb70e674be4917f3bcc5237c019, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681354431653 2023-04-13 02:53:53,626 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f6a62df11c3749358523995b9a58a865, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681354432801 2023-04-13 02:53:53,626 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 516e42b78e7f4120a889939bebc41066, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681354432801 2023-04-13 02:53:53,627 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 607bce681fb944e9a17964874de6fe91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433470 2023-04-13 02:53:53,628 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 59828948cccd4bf2a4e5e1e81aee4765, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433467 2023-04-13 02:53:53,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:53,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:53,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:53,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:53,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:53,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:53,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:53,653 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#63 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:53,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:53,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:53,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:53,674 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:53,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:53,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:53,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:53,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:53,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@834fc5c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:53,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:53,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:53,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:53,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:53,701 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:53,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:53,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:53,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:53,701 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:53,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,703 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354493696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354493704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,706 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:53,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:53,709 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,710 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:53,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,715 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/9fb4ea83b7aa460b9beedffddf5851e4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/9fb4ea83b7aa460b9beedffddf5851e4 2023-04-13 02:53:53,730 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 9fb4ea83b7aa460b9beedffddf5851e4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:53,730 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/5e9700baba78403986d2c9c6a3483d20 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5e9700baba78403986d2c9c6a3483d20 2023-04-13 02:53:53,730 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,730 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=9, startTime=1681354433615; duration=0sec 2023-04-13 02:53:53,730 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:53,731 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:53:53,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/b62852a3e52e45f483989600505c31d8 2023-04-13 02:53:53,742 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:53:53,742 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:53,743 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,743 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d4c39c0c392343068d9214f7df0d449e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=83.8 K 2023-04-13 02:53:53,744 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d4c39c0c392343068d9214f7df0d449e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681354429379 2023-04-13 02:53:53,745 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 67880cb78e3d402295e04d8e5e1f653f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354429505 2023-04-13 02:53:53,745 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c92bc618f51f44aa8c432091dfacc94b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=626, earliestPutTs=1681354429887 2023-04-13 02:53:53,746 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 919e59cfd69f46a09b072bb766394730, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1681354431026 2023-04-13 02:53:53,746 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d9fcccecbea640ca8223fbbb183f75c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1681354431653 2023-04-13 02:53:53,747 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e587e45406b64f548dde6e5a5b2db465, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681354432801 2023-04-13 02:53:53,748 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9182e0c0d64b44d782b843aea480868f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433470 2023-04-13 02:53:53,752 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into 5e9700baba78403986d2c9c6a3483d20(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:53,752 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,752 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=9, startTime=1681354433615; duration=0sec 2023-04-13 02:53:53,752 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354493760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,770 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:53,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/96d3cd97b7a341688859c7d3e89e5d6b 2023-04-13 02:53:53,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354493807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354493807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:53,819 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/72e215e128544722841ce3e9464cec0c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/72e215e128544722841ce3e9464cec0c 2023-04-13 02:53:53,826 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 72e215e128544722841ce3e9464cec0c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:53,826 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,826 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=9, startTime=1681354433615; duration=0sec 2023-04-13 02:53:53,826 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:53,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/051100db34554958895d3126ea05f85f 2023-04-13 02:53:53,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/b62852a3e52e45f483989600505c31d8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8 2023-04-13 02:53:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:53,836 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:53,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:53,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:53,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,838 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:53,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:53,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:53,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8, entries=200, sequenceid=718, filesize=14.3 K 2023-04-13 02:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:53,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:53,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:53,844 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:53,846 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,846 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/96d3cd97b7a341688859c7d3e89e5d6b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b 2023-04-13 02:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa18cd5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:53,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:53,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,850 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:53,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:53,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:53,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b, entries=150, sequenceid=718, filesize=11.9 K 2023-04-13 02:53:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/051100db34554958895d3126ea05f85f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f 2023-04-13 02:53:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:53,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:53,857 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:53,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:53,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:53,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:53,857 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:53,859 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,859 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,859 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,859 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f, entries=150, sequenceid=718, filesize=11.9 K 2023-04-13 02:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 23777bf909bb0929ddad8e9cb44ee33e in 208ms, sequenceid=718, compaction requested=false 2023-04-13 02:53:53,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:53,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:53,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:53,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:53,873 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,873 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:53,873 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:53,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:53,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:53,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:53,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:53,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:53,979 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:53,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:53,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:53,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354493979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:53,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:53,982 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,982 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:53,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:53,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:53,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354493981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:53,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:53,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:53,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:53,992 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,992 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:53,992 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:53,992 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:53,995 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1439b514[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:53,997 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:53,997 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:53,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:53,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:54,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:54,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,002 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:54,007 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:54,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:54,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:54,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:54,007 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:54,012 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,012 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,013 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354494012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354494013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,014 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:54,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:54,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:54,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:54,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354494067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354494082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354494084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:54,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:54,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:54,123 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:54,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:54,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:54,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:54,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:54,129 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,129 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:54,129 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:54,129 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:54,129 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:54,131 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c672eb8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:54,133 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:54,133 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:54,134 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:54,135 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,135 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:54,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:54,143 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:54,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:54,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:54,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:54,143 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:54,146 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,146 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,146 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:54,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:54,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:54,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:54,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:54,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:54,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:54,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:54,260 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:54,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:54,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:54,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,263 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:54,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:54,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:54,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:54,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:54,268 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:54,271 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,271 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:54,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339ee774[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:54,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:54,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:54,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:54,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:54,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,278 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:54,284 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:54,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:54,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:54,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:54,284 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:54,287 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,287 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,287 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354494287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354494288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:54,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:54,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:54,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:54,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:54,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:54,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,299 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:54,299 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:54,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:54,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354494315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354494317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/5889448f4e7c46b1bbeff8525a0d2e4c 2023-04-13 02:53:54,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/91c4443f1cc24552acf745779f4a0ecc 2023-04-13 02:53:54,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=739 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/25e1a9b3b35b42f989724baddcef4090 2023-04-13 02:53:54,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/5889448f4e7c46b1bbeff8525a0d2e4c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c 2023-04-13 02:53:54,399 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:54,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:54,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c, entries=150, sequenceid=739, filesize=11.9 K 2023-04-13 02:53:54,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:54,404 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/91c4443f1cc24552acf745779f4a0ecc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc 2023-04-13 02:53:54,409 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:54,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:54,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:54,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc, entries=100, sequenceid=739, filesize=9.5 K 2023-04-13 02:53:54,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:54,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/25e1a9b3b35b42f989724baddcef4090 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090 2023-04-13 02:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:54,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:54,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090, entries=100, sequenceid=739, filesize=9.5 K 2023-04-13 02:53:54,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 23777bf909bb0929ddad8e9cb44ee33e in 514ms, sequenceid=739, compaction requested=true 2023-04-13 02:53:54,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:54,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:54,455 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:54,455 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:54,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:54,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:54,456 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:54,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:54,457 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:54,457 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:54,457 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39710 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:54,457 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:54,457 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,457 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5e9700baba78403986d2c9c6a3483d20, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=38.8 K 2023-04-13 02:53:54,459 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 5e9700baba78403986d2c9c6a3483d20, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433470 2023-04-13 02:53:54,459 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:54,459 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:54,459 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,459 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/9fb4ea83b7aa460b9beedffddf5851e4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=34.0 K 2023-04-13 02:53:54,460 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b62852a3e52e45f483989600505c31d8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1681354433537 2023-04-13 02:53:54,460 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9fb4ea83b7aa460b9beedffddf5851e4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433470 2023-04-13 02:53:54,460 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 5889448f4e7c46b1bbeff8525a0d2e4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433694 2023-04-13 02:53:54,461 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 96d3cd97b7a341688859c7d3e89e5d6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1681354433537 2023-04-13 02:53:54,464 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 91c4443f1cc24552acf745779f4a0ecc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433694 2023-04-13 02:53:54,485 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:54,493 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#74 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:54,512 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/4f009229c9744bc6afab6ba7ba3a7cb6 2023-04-13 02:53:54,546 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/d87718bf561d474391494f9610aee31d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d87718bf561d474391494f9610aee31d 2023-04-13 02:53:54,557 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into d87718bf561d474391494f9610aee31d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:54,557 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:54,557 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=13, startTime=1681354434455; duration=0sec 2023-04-13 02:53:54,557 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:54,557 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:53:54,559 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:53:54,559 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:54,560 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:54,560 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/72e215e128544722841ce3e9464cec0c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=34.0 K 2023-04-13 02:53:54,560 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 72e215e128544722841ce3e9464cec0c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681354433470 2023-04-13 02:53:54,561 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 051100db34554958895d3126ea05f85f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1681354433537 2023-04-13 02:53:54,562 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 25e1a9b3b35b42f989724baddcef4090, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433694 2023-04-13 02:53:54,568 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/b91f40d56bc34e99b2c265a209d8b8c6 2023-04-13 02:53:54,574 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:54,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:54,604 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/6e21075d995a452aa64b18507433cc4f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6e21075d995a452aa64b18507433cc4f 2023-04-13 02:53:54,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354494602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52092 deadline: 1681354494603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354494606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,613 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 6e21075d995a452aa64b18507433cc4f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:54,613 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:54,613 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=13, startTime=1681354434455; duration=0sec 2023-04-13 02:53:54,614 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354494707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354494707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354494709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354494820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354494823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354494911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354494913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354494914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:54,933 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/358db4d7bc914ceeb5707ad9e49ba7be as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/358db4d7bc914ceeb5707ad9e49ba7be 2023-04-13 02:53:54,940 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 358db4d7bc914ceeb5707ad9e49ba7be(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:54,940 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:54,940 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=13, startTime=1681354434455; duration=0sec 2023-04-13 02:53:54,940 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:55,028 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/6741b870991d4f6ca22cc7076efa5fc2 2023-04-13 02:53:55,033 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/4f009229c9744bc6afab6ba7ba3a7cb6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6 2023-04-13 02:53:55,039 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6, entries=150, sequenceid=753, filesize=11.9 K 2023-04-13 02:53:55,040 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/b91f40d56bc34e99b2c265a209d8b8c6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6 2023-04-13 02:53:55,047 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6, entries=150, sequenceid=753, filesize=11.9 K 2023-04-13 02:53:55,048 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/6741b870991d4f6ca22cc7076efa5fc2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2 2023-04-13 02:53:55,052 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2, entries=150, sequenceid=753, filesize=11.9 K 2023-04-13 02:53:55,053 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 23777bf909bb0929ddad8e9cb44ee33e in 597ms, sequenceid=753, compaction requested=false 2023-04-13 02:53:55,053 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:55,053 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:55,055 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cf02f6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:55,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:55,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:55,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:55,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:55,062 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:55,065 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:55,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:55,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:55,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:55,065 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:55,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:55,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:55,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:55,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:55,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:55,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:55,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:55,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:55,072 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:55,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:55,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,072 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:55,172 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:55,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:55,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:55,175 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:55,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:55,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:55,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:55,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:55,177 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:55,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:55,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:55,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:55,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:55,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:55,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:55,179 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:55,180 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:55,180 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:55,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:55,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354495230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354495230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354495231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354495333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354495333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354495334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354495537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354495538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354495538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,592 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/f00112caeacd4ad49d8f54709e32eed6 2023-04-13 02:53:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354495828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354495831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354495838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354495842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:55,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354495842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,013 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/2e38060c107147f7b514e2c6b380fba4 2023-04-13 02:53:56,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:56,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354496343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354496345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354496350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,482 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/b9663f7338fb48b69cbcd90335349855 2023-04-13 02:53:56,495 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/f00112caeacd4ad49d8f54709e32eed6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6 2023-04-13 02:53:56,502 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 02:53:56,503 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/2e38060c107147f7b514e2c6b380fba4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4 2023-04-13 02:53:56,509 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 02:53:56,510 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/b9663f7338fb48b69cbcd90335349855 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855 2023-04-13 02:53:56,517 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 02:53:56,519 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 23777bf909bb0929ddad8e9cb44ee33e in 1338ms, sequenceid=774, compaction requested=true 2023-04-13 02:53:56,519 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:56,519 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:56,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:56,521 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:56,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c07924e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:56,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:56,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,524 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:56,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:56,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:56,531 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:56,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:56,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:56,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:56,531 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:56,533 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,533 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,533 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:56,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:56,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:56,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:56,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:56,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:56,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:56,646 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:56,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:56,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:56,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:56,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:56,649 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:56,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:56,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:56,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:56,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:56,652 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:56,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:56,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:56,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:56,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:56,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:56,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:56,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:56,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:56,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:56,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:56,654 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:56,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:57,063 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/640d9579681d478893c7f1a6afc4fbbc 2023-04-13 02:53:57,075 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/a6935ccbcede405185c0376e0a28868b 2023-04-13 02:53:57,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:57,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:57,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354497378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354497379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354497380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354497485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354497485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354497485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,488 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=788 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/724b0e12ccea4dc7922c99643b11a38d 2023-04-13 02:53:57,496 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/640d9579681d478893c7f1a6afc4fbbc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc 2023-04-13 02:53:57,504 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc, entries=150, sequenceid=788, filesize=11.9 K 2023-04-13 02:53:57,505 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/a6935ccbcede405185c0376e0a28868b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b 2023-04-13 02:53:57,515 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b, entries=150, sequenceid=788, filesize=11.9 K 2023-04-13 02:53:57,521 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/724b0e12ccea4dc7922c99643b11a38d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d 2023-04-13 02:53:57,526 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d, entries=150, sequenceid=788, filesize=11.9 K 2023-04-13 02:53:57,527 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 23777bf909bb0929ddad8e9cb44ee33e in 873ms, sequenceid=788, compaction requested=true 2023-04-13 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:57,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:57,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:57,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59307198[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:57,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:57,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:57,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:57,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:57,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:57,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:57,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:57,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:57,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:57,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:57,553 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:57,553 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:57,653 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:57,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:57,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:57,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:57,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:57,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:57,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:57,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 02:53:57,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354437378, max timestamp=1681354437691Num uniques 859; 2023-04-13 02:53:57,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354437378, max timestamp=1681354437691Num uniques 859; 2023-04-13 02:53:57,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:57,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:57,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:57,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:57,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354437378, max timestamp=1681354437691Num uniques 859; 2023-04-13 02:53:57,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:57,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d51888d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:57,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:57,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,708 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:57,709 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:57,709 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,709 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:57,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,711 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=809 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/c8c1b0e3e3064dc5a9ef519d0c32bf40 2023-04-13 02:53:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:57,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:57,716 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:57,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:57,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:57,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:57,716 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:57,718 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,718 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,721 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,725 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:57,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:57,728 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:57,728 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:57,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354437719, max timestamp=1681354437741Num uniques 859; 2023-04-13 02:53:57,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:57,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354437719, max timestamp=1681354437741Num uniques 859; 2023-04-13 02:53:57,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354437719, max timestamp=1681354437741Num uniques 859; 2023-04-13 02:53:57,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:57,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354437757, max timestamp=1681354437763Num uniques 1118; 2023-04-13 02:53:57,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354437757, max timestamp=1681354437763Num uniques 1118; 2023-04-13 02:53:57,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354497763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-13 02:53:57,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-13 02:53:57,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354437757, max timestamp=1681354437763Num uniques 1118; 2023-04-13 02:53:57,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-13 02:53:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354497796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681354437783, max timestamp=1681354437797Num uniques 1127; 2023-04-13 02:53:57,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-13 02:53:57,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681354437783, max timestamp=1681354437797Num uniques 1127; 2023-04-13 02:53:57,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681354437783, max timestamp=1681354437797Num uniques 1127; 2023-04-13 02:53:57,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-13 02:53:57,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-13 02:53:57,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:57,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:57,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:57,832 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:57,832 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,832 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:57,832 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:57,832 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:57,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:57,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:57,837 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,837 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:57,837 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:57,837 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:57,839 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4054e163[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:57,842 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:57,842 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,843 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:57,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:57,845 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:57,848 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:57,848 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:57,848 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:57,848 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:57,849 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:57,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681354437825, max timestamp=1681354437849Num uniques 1136; 2023-04-13 02:53:57,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-13 02:53:57,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681354437825, max timestamp=1681354437849Num uniques 1136; 2023-04-13 02:53:57,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-13 02:53:57,851 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681354437825, max timestamp=1681354437849Num uniques 1136; 2023-04-13 02:53:57,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-13 02:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,854 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:57,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:57,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:57,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:57,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:57,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354497870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354497874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681354437868, max timestamp=1681354437875Num uniques 1145; 2023-04-13 02:53:57,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-13 02:53:57,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13507, totalHeapSize=30920, min timestamp=1681354437868, max timestamp=1681354437876Num uniques 1145; 2023-04-13 02:53:57,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1681354437868, max timestamp=1681354437876Num uniques 1145; 2023-04-13 02:53:57,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=13500, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-13 02:53:57,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52437, new segment data size=13507, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-13 02:53:57,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354497903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354497909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354497910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:57,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:57,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:57,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:57,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:57,962 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:57,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:57,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:57,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,963 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,963 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:57,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:57,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:57,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:57,966 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:57,966 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:57,967 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:57,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ec5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:57,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:57,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:57,972 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,972 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:57,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:57,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:57,976 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:57,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:57,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:57,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:57,976 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:57,977 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,977 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,977 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:57,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,978 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,980 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:57,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:57,982 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:57,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:57,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:57,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:57,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:57,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:57,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:57,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354497979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:57,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:57,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354497980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354498012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354498014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354498014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,087 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,087 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,087 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,087 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,088 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,088 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,094 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,095 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,095 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae783ec[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,117 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,122 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,122 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:58,130 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,130 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=809 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/19c113a39be146aa9052d57d1d178836 2023-04-13 02:53:58,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=809 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d59d9a23e2ef40f185d570b23ce6e8eb 2023-04-13 02:53:58,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/c8c1b0e3e3064dc5a9ef519d0c32bf40 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40 2023-04-13 02:53:58,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354498185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354498185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40, entries=150, sequenceid=809, filesize=11.9 K 2023-04-13 02:53:58,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/19c113a39be146aa9052d57d1d178836 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836 2023-04-13 02:53:58,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836, entries=150, sequenceid=809, filesize=11.9 K 2023-04-13 02:53:58,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d59d9a23e2ef40f185d570b23ce6e8eb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb 2023-04-13 02:53:58,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb, entries=150, sequenceid=809, filesize=11.9 K 2023-04-13 02:53:58,215 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=154.31 KB/158010 for 23777bf909bb0929ddad8e9cb44ee33e in 524ms, sequenceid=809, compaction requested=true 2023-04-13 02:53:58,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:58,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:58,215 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:53:58,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:58,215 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:53:58,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:58,218 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:53:58,218 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:53:58,218 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,218 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/358db4d7bc914ceeb5707ad9e49ba7be, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=60.3 K 2023-04-13 02:53:58,218 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:53:58,218 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:53:58,219 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,219 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d87718bf561d474391494f9610aee31d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=60.3 K 2023-04-13 02:53:58,219 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 358db4d7bc914ceeb5707ad9e49ba7be, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433542 2023-04-13 02:53:58,219 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d87718bf561d474391494f9610aee31d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433542 2023-04-13 02:53:58,219 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b91f40d56bc34e99b2c265a209d8b8c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681354433973 2023-04-13 02:53:58,220 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4f009229c9744bc6afab6ba7ba3a7cb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681354433973 2023-04-13 02:53:58,220 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2e38060c107147f7b514e2c6b380fba4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1681354434594 2023-04-13 02:53:58,220 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f00112caeacd4ad49d8f54709e32eed6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1681354434594 2023-04-13 02:53:58,220 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a6935ccbcede405185c0376e0a28868b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1681354435225 2023-04-13 02:53:58,221 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 640d9579681d478893c7f1a6afc4fbbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1681354435225 2023-04-13 02:53:58,221 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 19c113a39be146aa9052d57d1d178836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:53:58,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354437902, max timestamp=1681354438221Num uniques 1154; 2023-04-13 02:53:58,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-13 02:53:58,223 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c8c1b0e3e3064dc5a9ef519d0c32bf40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:53:58,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354437902, max timestamp=1681354438221Num uniques 1154; 2023-04-13 02:53:58,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-13 02:53:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:58,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=71.56 KB heapSize=195.34 KB 2023-04-13 02:53:58,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681354437902, max timestamp=1681354438221Num uniques 1154; 2023-04-13 02:53:58,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:58,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:58,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,238 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,241 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#88 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:58,243 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#89 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:58,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354438245, max timestamp=1681354438255Num uniques 859; 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354438245, max timestamp=1681354438255Num uniques 859; 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354438245, max timestamp=1681354438255Num uniques 859; 2023-04-13 02:53:58,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b637ddd[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,266 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/e01bda013c0c4495933bee6637a2bd0d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e01bda013c0c4495933bee6637a2bd0d 2023-04-13 02:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,268 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,268 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,268 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,268 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,268 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,272 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into e01bda013c0c4495933bee6637a2bd0d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:58,272 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:58,272 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=11, startTime=1681354438215; duration=0sec 2023-04-13 02:53:58,273 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:53:58,273 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:53:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,275 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:53:58,275 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,275 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,276 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6e21075d995a452aa64b18507433cc4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=60.3 K 2023-04-13 02:53:58,276 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6e21075d995a452aa64b18507433cc4f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=739, earliestPutTs=1681354433542 2023-04-13 02:53:58,276 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6741b870991d4f6ca22cc7076efa5fc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681354433973 2023-04-13 02:53:58,277 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b9663f7338fb48b69cbcd90335349855, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1681354434594 2023-04-13 02:53:58,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,277 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 724b0e12ccea4dc7922c99643b11a38d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=788, earliestPutTs=1681354435225 2023-04-13 02:53:58,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:58,278 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d59d9a23e2ef40f185d570b23ce6e8eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:53:58,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,278 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354498283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354438272, max timestamp=1681354438290Num uniques 1118; 2023-04-13 02:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-13 02:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354438272, max timestamp=1681354438290Num uniques 1118; 2023-04-13 02:53:58,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-13 02:53:58,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354438272, max timestamp=1681354438290Num uniques 1118; 2023-04-13 02:53:58,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-13 02:53:58,298 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:53:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354498333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354498333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,378 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,383 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,383 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,383 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,383 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,384 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,384 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,386 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,389 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,389 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,389 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,389 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,393 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354498397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5bc22b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,420 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,420 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,436 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,436 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354498441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354498441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354498490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354498493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,541 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,543 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,543 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202006c3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,560 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,560 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,563 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,571 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,571 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:58,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,572 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,572 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354498599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354498645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354498646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=917 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/73c9f02b65cc453c8875542ac9e89c5c 2023-04-13 02:53:58,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=917 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/314c1e99217f41fba5c4372e41736f6a 2023-04-13 02:53:58,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/28e03642f1244fc0aa0d2c1a33999380 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/28e03642f1244fc0aa0d2c1a33999380 2023-04-13 02:53:58,672 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,680 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 28e03642f1244fc0aa0d2c1a33999380(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:58,680 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:58,680 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=11, startTime=1681354438215; duration=0sec 2023-04-13 02:53:58,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,680 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:58,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,680 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46be09c8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,694 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,694 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,697 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,700 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:58,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,709 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/1af10a37a8e344f2a0e8571bb1eaf38d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1af10a37a8e344f2a0e8571bb1eaf38d 2023-04-13 02:53:58,715 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 1af10a37a8e344f2a0e8571bb1eaf38d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:53:58,715 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:58,715 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=11, startTime=1681354438215; duration=0sec 2023-04-13 02:53:58,715 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:53:58,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,812 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,818 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,818 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,818 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,818 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,820 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e30957[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,822 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,826 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,831 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,831 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,831 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,833 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,833 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,833 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,841 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,841 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354498909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:58,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:58,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:58,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:58,945 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:58,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:58,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:58,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:58,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:58,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:58,949 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,949 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:58,949 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:58,949 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:58,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354498949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354498950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:58,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dbdb3a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:58,954 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:58,954 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:58,956 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,956 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:58,957 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:58,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:58,960 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:58,960 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:58,960 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:58,960 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:58,960 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:58,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:58,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:58,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:58,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:58,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:58,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:58,970 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:58,970 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:58,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354498995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354498999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,070 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:59,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:59,074 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:59,074 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,074 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:59,074 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:59,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:59,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:59,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:59,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:59,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446ddf7c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:59,083 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:59,083 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:59,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=917 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/07be69f571a24f60bc7bd41c4588ef36 2023-04-13 02:53:59,086 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,086 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:59,088 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:59,096 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:59,096 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:59,096 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:59,096 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:59,096 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:59,099 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,099 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/73c9f02b65cc453c8875542ac9e89c5c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c 2023-04-13 02:53:59,099 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c, entries=250, sequenceid=917, filesize=16.7 K 2023-04-13 02:53:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:59,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:59,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:59,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:59,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/314c1e99217f41fba5c4372e41736f6a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a 2023-04-13 02:53:59,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a, entries=150, sequenceid=917, filesize=11.9 K 2023-04-13 02:53:59,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/07be69f571a24f60bc7bd41c4588ef36 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36 2023-04-13 02:53:59,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36, entries=150, sequenceid=917, filesize=11.9 K 2023-04-13 02:53:59,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.58 KB/88656, currentSize=154.31 KB/158010 for 23777bf909bb0929ddad8e9cb44ee33e in 902ms, sequenceid=917, compaction requested=false 2023-04-13 02:53:59,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:59,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:59,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:59,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:59,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:59,213 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:59,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:59,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:59,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,215 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,215 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,215 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,215 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:59,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:59,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:59,217 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:59,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:53:59,228 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/2c79d760033e436985de6969ad16dc83 2023-04-13 02:53:59,271 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/bc62d1e6e65a45668157b769292b4ed2 2023-04-13 02:53:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:59,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:59,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354499428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354499457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354499457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354499533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,688 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2076c9fc727c474d82998a08f797c0ea 2023-04-13 02:53:59,695 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/2c79d760033e436985de6969ad16dc83 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83 2023-04-13 02:53:59,700 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83, entries=150, sequenceid=974, filesize=11.9 K 2023-04-13 02:53:59,701 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/bc62d1e6e65a45668157b769292b4ed2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2 2023-04-13 02:53:59,705 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2, entries=150, sequenceid=974, filesize=11.9 K 2023-04-13 02:53:59,706 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2076c9fc727c474d82998a08f797c0ea as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea 2023-04-13 02:53:59,711 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea, entries=150, sequenceid=974, filesize=11.9 K 2023-04-13 02:53:59,711 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for 23777bf909bb0929ddad8e9cb44ee33e in 494ms, sequenceid=974, compaction requested=true 2023-04-13 02:53:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:53:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,715 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ab7e70[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:59,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:59,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:59,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,720 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:59,721 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:59,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:59,723 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:59,723 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:59,723 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:59,723 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:59,723 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:59,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,725 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,730 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:59,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:59,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,731 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:59,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:59,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:59,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:53:59,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:53:59,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:53:59,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:59,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:53:59,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:59,742 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:53:59,742 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:53:59,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/ed6f23ba8cb5419487c4a56990b36911 2023-04-13 02:53:59,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354439781, max timestamp=1681354439796Num uniques 859; 2023-04-13 02:53:59,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:59,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354439781, max timestamp=1681354439796Num uniques 859; 2023-04-13 02:53:59,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354439781, max timestamp=1681354439796Num uniques 859; 2023-04-13 02:53:59,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:59,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:53:59,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:59,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:59,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:59,834 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:59,834 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,834 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:59,834 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:59,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354499833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,837 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:59,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:59,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:59,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:59,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:59,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:59,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:59,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:59,843 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:59,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0c78c3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:59,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:59,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,846 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,846 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:59,846 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,846 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:59,846 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:59,846 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:59,848 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,848 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:59,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:59,850 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:59,850 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:59,853 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,853 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,853 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,854 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:59,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,858 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,858 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:59,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:53:59,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:53:59,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:53:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354499936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:53:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:53:59,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:53:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:53:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:53:59,961 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:53:59,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:53:59,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:53:59,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,969 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,970 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:53:59,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:53:59,971 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:53:59,972 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,972 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:53:59,972 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:53:59,972 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:53:59,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:53:59,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:53:59,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b46e73[Count = 0] remaining members to acquire global barrier 2023-04-13 02:53:59,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:53:59,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,977 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,977 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:53:59,977 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:53:59,977 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:53:59,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:53:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:53:59,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:53:59,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:53:59,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:53:59,992 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:53:59,992 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:53:59,992 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:53:59,992 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:53:59,992 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:53:59,994 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,994 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,994 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:53:59,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:53:59,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,998 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:53:59,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:53:59,999 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:53:59,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:53:59,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:53:59,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:53:59,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:53:59,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:53:59,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:00,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:00,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52076 deadline: 1681354500006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52106 deadline: 1681354500006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,100 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:00,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:00,103 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:00,103 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:00,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:00,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:00,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,106 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:00,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:00,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:54:00,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:54:00,115 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:00,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,117 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,118 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:00,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c88cfe6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:00,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:00,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:00,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:00,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:00,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:00,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:00,127 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:00,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:00,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:00,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:00,127 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:00,129 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,129 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:00,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:00,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,133 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:00,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:00,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:00,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:00,137 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:00,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:00,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:00,137 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:00,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:00,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354500141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/8626aa7d61b24e38854e0a15c8b79bd5 2023-04-13 02:54:00,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2c071c0bfd534a2db96185dd5ccad643 2023-04-13 02:54:00,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/ed6f23ba8cb5419487c4a56990b36911 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911 2023-04-13 02:54:00,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911, entries=200, sequenceid=984, filesize=14.3 K 2023-04-13 02:54:00,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/8626aa7d61b24e38854e0a15c8b79bd5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5 2023-04-13 02:54:00,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5, entries=150, sequenceid=984, filesize=11.9 K 2023-04-13 02:54:00,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2c071c0bfd534a2db96185dd5ccad643 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643 2023-04-13 02:54:00,237 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:00,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:00,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:00,243 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:00,243 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:00,243 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:00,243 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:00,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643, entries=150, sequenceid=984, filesize=11.9 K 2023-04-13 02:54:00,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 23777bf909bb0929ddad8e9cb44ee33e in 504ms, sequenceid=984, compaction requested=true 2023-04-13 02:54:00,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:00,245 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:00,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:00,245 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:00,245 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:00,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:00,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:00,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:00,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:00,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:00,248 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57052 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:00,248 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/A is initiating minor compaction (all files) 2023-04-13 02:54:00,248 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/A in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,248 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e01bda013c0c4495933bee6637a2bd0d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=55.7 K 2023-04-13 02:54:00,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:00,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/B is initiating minor compaction (all files) 2023-04-13 02:54:00,248 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/B in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,248 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/28e03642f1244fc0aa0d2c1a33999380, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=48.6 K 2023-04-13 02:54:00,249 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e01bda013c0c4495933bee6637a2bd0d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:54:00,249 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 28e03642f1244fc0aa0d2c1a33999380, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:54:00,249 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 73c9f02b65cc453c8875542ac9e89c5c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=917, earliestPutTs=1681354437902 2023-04-13 02:54:00,250 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 314c1e99217f41fba5c4372e41736f6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=917, earliestPutTs=1681354437902 2023-04-13 02:54:00,250 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2c79d760033e436985de6969ad16dc83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681354438312 2023-04-13 02:54:00,250 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting bc62d1e6e65a45668157b769292b4ed2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681354438312 2023-04-13 02:54:00,250 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ed6f23ba8cb5419487c4a56990b36911, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681354439416 2023-04-13 02:54:00,251 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8626aa7d61b24e38854e0a15c8b79bd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681354439423 2023-04-13 02:54:00,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:00,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. started... 2023-04-13 02:54:00,257 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:54:00,257 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:00,260 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#B#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:00,264 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#A#compaction#100 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:00,283 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/1188dd18aa6745e7b2c9c771adf8af3d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/1188dd18aa6745e7b2c9c771adf8af3d 2023-04-13 02:54:00,288 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/A of 23777bf909bb0929ddad8e9cb44ee33e into 1188dd18aa6745e7b2c9c771adf8af3d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:00,288 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:00,288 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/A, priority=12, startTime=1681354440245; duration=0sec 2023-04-13 02:54:00,288 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:00,288 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:00,290 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:00,290 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 23777bf909bb0929ddad8e9cb44ee33e/C is initiating minor compaction (all files) 2023-04-13 02:54:00,290 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 23777bf909bb0929ddad8e9cb44ee33e/C in TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:00,290 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1af10a37a8e344f2a0e8571bb1eaf38d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp, totalSize=48.6 K 2023-04-13 02:54:00,291 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 1af10a37a8e344f2a0e8571bb1eaf38d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=809, earliestPutTs=1681354437379 2023-04-13 02:54:00,293 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 07be69f571a24f60bc7bd41c4588ef36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=917, earliestPutTs=1681354437902 2023-04-13 02:54:00,293 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2076c9fc727c474d82998a08f797c0ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681354438312 2023-04-13 02:54:00,293 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2c071c0bfd534a2db96185dd5ccad643, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681354439423 2023-04-13 02:54:00,301 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 23777bf909bb0929ddad8e9cb44ee33e#C#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:00,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. as already flushing 2023-04-13 02:54:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52092 deadline: 1681354500462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354500464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52072 deadline: 1681354500468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354500568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:00,676 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/78c2cfe3f7704ae992a4282210ca01d6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/78c2cfe3f7704ae992a4282210ca01d6 2023-04-13 02:54:00,680 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/9e540e00e38e4f3a8333868322c5d7de 2023-04-13 02:54:00,683 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/B of 23777bf909bb0929ddad8e9cb44ee33e into 78c2cfe3f7704ae992a4282210ca01d6(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:00,684 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:00,684 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/B, priority=12, startTime=1681354440245; duration=0sec 2023-04-13 02:54:00,684 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:00,730 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/61a77699a6c049a5b8fb5a502a4673a9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/61a77699a6c049a5b8fb5a502a4673a9 2023-04-13 02:54:00,736 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 23777bf909bb0929ddad8e9cb44ee33e/C of 23777bf909bb0929ddad8e9cb44ee33e into 61a77699a6c049a5b8fb5a502a4673a9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:00,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:00,736 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e., storeName=23777bf909bb0929ddad8e9cb44ee33e/C, priority=12, startTime=1681354440246; duration=0sec 2023-04-13 02:54:00,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:00,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:00,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354500779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:01,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52070 deadline: 1681354501083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,099 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f3c960f4eba945b2a9f6772215cdc809 2023-04-13 02:54:01,349 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00375c9d to 127.0.0.1:62948 2023-04-13 02:54:01,349 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:01,349 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5979620a to 127.0.0.1:62948 2023-04-13 02:54:01,349 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:01,354 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ec2fae8 to 127.0.0.1:62948 2023-04-13 02:54:01,354 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27685d29 to 127.0.0.1:62948 2023-04-13 02:54:01,354 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:01,354 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:01,517 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/e24533fd480f46998f851acb3a8d77df 2023-04-13 02:54:01,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/9e540e00e38e4f3a8333868322c5d7de as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e540e00e38e4f3a8333868322c5d7de 2023-04-13 02:54:01,528 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e540e00e38e4f3a8333868322c5d7de, entries=150, sequenceid=1020, filesize=11.9 K 2023-04-13 02:54:01,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/f3c960f4eba945b2a9f6772215cdc809 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f3c960f4eba945b2a9f6772215cdc809 2023-04-13 02:54:01,534 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f3c960f4eba945b2a9f6772215cdc809, entries=150, sequenceid=1020, filesize=11.9 K 2023-04-13 02:54:01,535 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/e24533fd480f46998f851acb3a8d77df as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e24533fd480f46998f851acb3a8d77df 2023-04-13 02:54:01,540 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e24533fd480f46998f851acb3a8d77df, entries=150, sequenceid=1020, filesize=11.9 K 2023-04-13 02:54:01,541 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 23777bf909bb0929ddad8e9cb44ee33e in 1285ms, sequenceid=1020, compaction requested=false 2023-04-13 02:54:01,541 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:01,541 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:01,541 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:01,541 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:01,541 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,541 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:01,541 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:01,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457974f9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:01,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:01,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:01,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:01,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:01,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:01,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:01,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:01,551 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:01,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:01,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:01,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:01,552 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:01,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:01,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:01,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:01,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:01,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:01,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:01,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:01,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:01,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:01,560 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:01,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:01,560 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:01,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:01,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:01,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:01,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:01,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:01,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:54:01,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:01,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:54:01,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:01,592 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72ea4a65 to 127.0.0.1:62948 2023-04-13 02:54:01,592 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:01,660 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:01,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:02,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/172b23f0524449e2ac37534fbd1d2006 2023-04-13 02:54:02,027 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1645b6b0 to 127.0.0.1:62948 2023-04-13 02:54:02,027 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:02,029 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b52aa42 to 127.0.0.1:62948 2023-04-13 02:54:02,029 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:02,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/e53ef812af954ba2acd5544bd97530eb 2023-04-13 02:54:02,476 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1710407d to 127.0.0.1:62948 2023-04-13 02:54:02,477 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:02,486 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bc6d729 to 127.0.0.1:62948 2023-04-13 02:54:02,487 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 257 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 189 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 3933 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 3962 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1772 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5312 rows 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1779 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5336 rows 2023-04-13 02:54:02,487 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:54:02,488 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d4f988c to 127.0.0.1:62948 2023-04-13 02:54:02,488 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:02,492 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:54:02,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:54:02,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:02,507 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354442507"}]},"ts":"1681354442507"} 2023-04-13 02:54:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:02,508 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:54:02,511 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:54:02,512 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, UNASSIGN}] 2023-04-13 02:54:02,514 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, UNASSIGN 2023-04-13 02:54:02,514 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=23777bf909bb0929ddad8e9cb44ee33e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:02,515 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354442514"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354442514"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354442514"}]},"ts":"1681354442514"} 2023-04-13 02:54:02,516 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:02,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:02,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:02,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 23777bf909bb0929ddad8e9cb44ee33e, disabling compactions & flushes 2023-04-13 02:54:02,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:02,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d74c351eb9eb4f9885a52a9eb1f17e43 2023-04-13 02:54:02,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/172b23f0524449e2ac37534fbd1d2006 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/172b23f0524449e2ac37534fbd1d2006 2023-04-13 02:54:02,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/172b23f0524449e2ac37534fbd1d2006, entries=150, sequenceid=1034, filesize=11.9 K 2023-04-13 02:54:02,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/e53ef812af954ba2acd5544bd97530eb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/e53ef812af954ba2acd5544bd97530eb 2023-04-13 02:54:02,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/e53ef812af954ba2acd5544bd97530eb, entries=150, sequenceid=1034, filesize=11.9 K 2023-04-13 02:54:02,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/d74c351eb9eb4f9885a52a9eb1f17e43 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d74c351eb9eb4f9885a52a9eb1f17e43 2023-04-13 02:54:02,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d74c351eb9eb4f9885a52a9eb1f17e43, entries=150, sequenceid=1034, filesize=11.9 K 2023-04-13 02:54:02,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 23777bf909bb0929ddad8e9cb44ee33e in 1273ms, sequenceid=1034, compaction requested=true 2023-04-13 02:54:02,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:02,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:02,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:02,864 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. because compaction request was cancelled 2023-04-13 02:54:02,864 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. because compaction request was cancelled 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:02,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. after waiting 0 ms 2023-04-13 02:54:02,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:02,864 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. because compaction request was cancelled 2023-04-13 02:54:02,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 23777bf909bb0929ddad8e9cb44ee33e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=A 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=B 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 23777bf909bb0929ddad8e9cb44ee33e, store=C 2023-04-13 02:54:02,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:03,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:03,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1041 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/aa7ab1fef961444eb98930c23e0f6d31 2023-04-13 02:54:03,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:03,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1041 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/20c6c231aeb14bd6810480200a13d0cc 2023-04-13 02:54:04,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1041 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2725befa80574134afde8f6561994d35 2023-04-13 02:54:04,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/A/aa7ab1fef961444eb98930c23e0f6d31 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aa7ab1fef961444eb98930c23e0f6d31 2023-04-13 02:54:04,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aa7ab1fef961444eb98930c23e0f6d31, entries=100, sequenceid=1041, filesize=9.5 K 2023-04-13 02:54:04,102 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/B/20c6c231aeb14bd6810480200a13d0cc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/20c6c231aeb14bd6810480200a13d0cc 2023-04-13 02:54:04,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/20c6c231aeb14bd6810480200a13d0cc, entries=100, sequenceid=1041, filesize=9.5 K 2023-04-13 02:54:04,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/.tmp/C/2725befa80574134afde8f6561994d35 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2725befa80574134afde8f6561994d35 2023-04-13 02:54:04,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2725befa80574134afde8f6561994d35, entries=100, sequenceid=1041, filesize=9.5 K 2023-04-13 02:54:04,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 23777bf909bb0929ddad8e9cb44ee33e in 1248ms, sequenceid=1041, compaction requested=true 2023-04-13 02:54:04,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e90404b4187644e4a15f19fe68ca42eb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2d0aa2a71b394232b290f6095c159804, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4937f22d36ea40ca9d38efb2d1293841, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/04ff373b0b144c38803928620eae059e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5e9700baba78403986d2c9c6a3483d20, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d87718bf561d474391494f9610aee31d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e01bda013c0c4495933bee6637a2bd0d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911] to archive 2023-04-13 02:54:04,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:04,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d1c7b314755b497297f7e430f1286862 2023-04-13 02:54:04,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e77d80f620b9410e8e4e3d2cd08915b3 2023-04-13 02:54:04,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/7fcabd81360b4ef5995bc183ecdb1d2b 2023-04-13 02:54:04,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e90404b4187644e4a15f19fe68ca42eb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e90404b4187644e4a15f19fe68ca42eb 2023-04-13 02:54:04,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/103fa7f8c8eb40d5ba2c518ef09946ed 2023-04-13 02:54:04,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/96dc41ca5f93447d9af9ae57e85a659b 2023-04-13 02:54:04,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2d0aa2a71b394232b290f6095c159804 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2d0aa2a71b394232b290f6095c159804 2023-04-13 02:54:04,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/254a2f2151ba4064bd2532cba37da7fb 2023-04-13 02:54:04,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aeac256c6a34458e94b8e116385c5dd4 2023-04-13 02:54:04,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e68d6d84a46448b6a966c70918f341be 2023-04-13 02:54:04,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4937f22d36ea40ca9d38efb2d1293841 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4937f22d36ea40ca9d38efb2d1293841 2023-04-13 02:54:04,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b3d688fd21404922b08f56bd477d4645 2023-04-13 02:54:04,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/12c838d01e7b48b2b0d1ae6695e8bd18 2023-04-13 02:54:04,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e803e3c971b461994b78412c4acb36b 2023-04-13 02:54:04,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/04ff373b0b144c38803928620eae059e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/04ff373b0b144c38803928620eae059e 2023-04-13 02:54:04,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/db54ed5554e044d18fc5b99a14e47ba3 2023-04-13 02:54:04,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/584ad7689df74710841beb49324e1ebe 2023-04-13 02:54:04,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d5eef4bd56ae40ee9998a7667a9fa751 2023-04-13 02:54:04,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/0c10a8e9bbe3472eb08f7da56996d681 2023-04-13 02:54:04,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f6a62df11c3749358523995b9a58a865 2023-04-13 02:54:04,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/59828948cccd4bf2a4e5e1e81aee4765 2023-04-13 02:54:04,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5e9700baba78403986d2c9c6a3483d20 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5e9700baba78403986d2c9c6a3483d20 2023-04-13 02:54:04,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/b62852a3e52e45f483989600505c31d8 2023-04-13 02:54:04,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d87718bf561d474391494f9610aee31d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/d87718bf561d474391494f9610aee31d 2023-04-13 02:54:04,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/5889448f4e7c46b1bbeff8525a0d2e4c 2023-04-13 02:54:04,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/4f009229c9744bc6afab6ba7ba3a7cb6 2023-04-13 02:54:04,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/f00112caeacd4ad49d8f54709e32eed6 2023-04-13 02:54:04,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/640d9579681d478893c7f1a6afc4fbbc 2023-04-13 02:54:04,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e01bda013c0c4495933bee6637a2bd0d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/e01bda013c0c4495933bee6637a2bd0d 2023-04-13 02:54:04,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/c8c1b0e3e3064dc5a9ef519d0c32bf40 2023-04-13 02:54:04,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/73c9f02b65cc453c8875542ac9e89c5c 2023-04-13 02:54:04,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/2c79d760033e436985de6969ad16dc83 2023-04-13 02:54:04,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/ed6f23ba8cb5419487c4a56990b36911 2023-04-13 02:54:04,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/53a3077d55854b11ac3d9d33af0cf85f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/7b47be0c018540918dcd7bc49140c400, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/0104fa72daa649cdaa6d92ceff35479f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/ce6ea08fe5e9447aa5aa526a86fae079, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/9fb4ea83b7aa460b9beedffddf5851e4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/358db4d7bc914ceeb5707ad9e49ba7be, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/28e03642f1244fc0aa0d2c1a33999380, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5] to archive 2023-04-13 02:54:04,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:04,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/89304eca7d6b4cd0af7c3f7ff30362dd 2023-04-13 02:54:04,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/caa6b11ad3794397b56094efbac183ab 2023-04-13 02:54:04,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/53a3077d55854b11ac3d9d33af0cf85f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/53a3077d55854b11ac3d9d33af0cf85f 2023-04-13 02:54:04,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/208ff3d80e494d5c8cac5b2612b97d7e 2023-04-13 02:54:04,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d74eb3da9d13472dae45afadcab7ece7 2023-04-13 02:54:04,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/7b47be0c018540918dcd7bc49140c400 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/7b47be0c018540918dcd7bc49140c400 2023-04-13 02:54:04,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8af79007fa604b5eacf1ad7e60e50be0 2023-04-13 02:54:04,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/1eb1ebe7a74943d5b489542a0d3cae66 2023-04-13 02:54:04,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/d305af5b5b424739a41fb2876cf4282c 2023-04-13 02:54:04,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/0104fa72daa649cdaa6d92ceff35479f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/0104fa72daa649cdaa6d92ceff35479f 2023-04-13 02:54:04,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f9e93af853f84001bfdb7b5e5cefe092 2023-04-13 02:54:04,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3d8b1b74ca3545439a9d289caa21d20a 2023-04-13 02:54:04,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f26af0891d4e4d8586de55ae71a11186 2023-04-13 02:54:04,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/ce6ea08fe5e9447aa5aa526a86fae079 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/ce6ea08fe5e9447aa5aa526a86fae079 2023-04-13 02:54:04,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/fbde30cf9e744bdaa4040c3b33e7a5c1 2023-04-13 02:54:04,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/09d5d51ccdbe451aaafa3c709e9632e2 2023-04-13 02:54:04,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/3ca013d92947430294012cd70246054d 2023-04-13 02:54:04,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/afa96c83d0274723b92e2322c316aa83 2023-04-13 02:54:04,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a7b01bb70e674be4917f3bcc5237c019 2023-04-13 02:54:04,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/516e42b78e7f4120a889939bebc41066 2023-04-13 02:54:04,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/9fb4ea83b7aa460b9beedffddf5851e4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/9fb4ea83b7aa460b9beedffddf5851e4 2023-04-13 02:54:04,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/607bce681fb944e9a17964874de6fe91 2023-04-13 02:54:04,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/96d3cd97b7a341688859c7d3e89e5d6b 2023-04-13 02:54:04,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/358db4d7bc914ceeb5707ad9e49ba7be to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/358db4d7bc914ceeb5707ad9e49ba7be 2023-04-13 02:54:04,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/91c4443f1cc24552acf745779f4a0ecc 2023-04-13 02:54:04,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/b91f40d56bc34e99b2c265a209d8b8c6 2023-04-13 02:54:04,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/2e38060c107147f7b514e2c6b380fba4 2023-04-13 02:54:04,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/a6935ccbcede405185c0376e0a28868b 2023-04-13 02:54:04,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/28e03642f1244fc0aa0d2c1a33999380 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/28e03642f1244fc0aa0d2c1a33999380 2023-04-13 02:54:04,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/19c113a39be146aa9052d57d1d178836 2023-04-13 02:54:04,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/314c1e99217f41fba5c4372e41736f6a 2023-04-13 02:54:04,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/bc62d1e6e65a45668157b769292b4ed2 2023-04-13 02:54:04,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/8626aa7d61b24e38854e0a15c8b79bd5 2023-04-13 02:54:04,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23502074dbae4bb6a7225ecc9e03e9c9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23e6753790a74209a506f8ee44d53650, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/edbb690fa77c47088383d00f8a03e30f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d4c39c0c392343068d9214f7df0d449e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/72e215e128544722841ce3e9464cec0c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6e21075d995a452aa64b18507433cc4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1af10a37a8e344f2a0e8571bb1eaf38d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643] to archive 2023-04-13 02:54:04,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:04,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/8ecdc06b82404d80b78712fff33f1a0f 2023-04-13 02:54:04,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6f29182a1bd443e88196dc4b6b25f617 2023-04-13 02:54:04,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23502074dbae4bb6a7225ecc9e03e9c9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23502074dbae4bb6a7225ecc9e03e9c9 2023-04-13 02:54:04,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/849d7f335c7c40cfa1c4b603635d64f1 2023-04-13 02:54:04,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/f6fc36966c344d808674c3473b99db6c 2023-04-13 02:54:04,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23e6753790a74209a506f8ee44d53650 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/23e6753790a74209a506f8ee44d53650 2023-04-13 02:54:04,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1da447461457452487448bb4dba5032c 2023-04-13 02:54:04,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/79086c5ff2f64faea00c797353d8ecb7 2023-04-13 02:54:04,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b70ebde0db39466c86cb338104cfd4f0 2023-04-13 02:54:04,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/edbb690fa77c47088383d00f8a03e30f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/edbb690fa77c47088383d00f8a03e30f 2023-04-13 02:54:04,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/644296831c5a4201a766f8e2fb42c741 2023-04-13 02:54:04,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/cfa3e68d41b641e8aff99eba2b511bd4 2023-04-13 02:54:04,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c74a674d59c6460f8a9728b7d47adbbb 2023-04-13 02:54:04,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d4c39c0c392343068d9214f7df0d449e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d4c39c0c392343068d9214f7df0d449e 2023-04-13 02:54:04,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/bdc14121925142d6a7ef745071db5552 2023-04-13 02:54:04,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/67880cb78e3d402295e04d8e5e1f653f 2023-04-13 02:54:04,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/c92bc618f51f44aa8c432091dfacc94b 2023-04-13 02:54:04,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/919e59cfd69f46a09b072bb766394730 2023-04-13 02:54:04,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d9fcccecbea640ca8223fbbb183f75c8 2023-04-13 02:54:04,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e587e45406b64f548dde6e5a5b2db465 2023-04-13 02:54:04,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/72e215e128544722841ce3e9464cec0c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/72e215e128544722841ce3e9464cec0c 2023-04-13 02:54:04,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/9182e0c0d64b44d782b843aea480868f 2023-04-13 02:54:04,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/051100db34554958895d3126ea05f85f 2023-04-13 02:54:04,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6e21075d995a452aa64b18507433cc4f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6e21075d995a452aa64b18507433cc4f 2023-04-13 02:54:04,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/25e1a9b3b35b42f989724baddcef4090 2023-04-13 02:54:04,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/6741b870991d4f6ca22cc7076efa5fc2 2023-04-13 02:54:04,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/b9663f7338fb48b69cbcd90335349855 2023-04-13 02:54:04,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/724b0e12ccea4dc7922c99643b11a38d 2023-04-13 02:54:04,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1af10a37a8e344f2a0e8571bb1eaf38d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/1af10a37a8e344f2a0e8571bb1eaf38d 2023-04-13 02:54:04,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d59d9a23e2ef40f185d570b23ce6e8eb 2023-04-13 02:54:04,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/07be69f571a24f60bc7bd41c4588ef36 2023-04-13 02:54:04,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2076c9fc727c474d82998a08f797c0ea 2023-04-13 02:54:04,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2c071c0bfd534a2db96185dd5ccad643 2023-04-13 02:54:04,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/recovered.edits/1044.seqid, newMaxSeqId=1044, maxSeqId=1 2023-04-13 02:54:04,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e. 2023-04-13 02:54:04,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 23777bf909bb0929ddad8e9cb44ee33e: 2023-04-13 02:54:04,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:04,288 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=23777bf909bb0929ddad8e9cb44ee33e, regionState=CLOSED 2023-04-13 02:54:04,288 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354444288"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354444288"}]},"ts":"1681354444288"} 2023-04-13 02:54:04,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-13 02:54:04,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 23777bf909bb0929ddad8e9cb44ee33e, server=jenkins-hbase4.apache.org,34919,1681354416534 in 1.7730 sec 2023-04-13 02:54:04,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-13 02:54:04,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=23777bf909bb0929ddad8e9cb44ee33e, UNASSIGN in 1.7790 sec 2023-04-13 02:54:04,294 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354444294"}]},"ts":"1681354444294"} 2023-04-13 02:54:04,295 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:54:04,297 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:54:04,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7980 sec 2023-04-13 02:54:04,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 02:54:04,613 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-13 02:54:04,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:54:04,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,625 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,627 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 02:54:04,632 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:04,635 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/recovered.edits] 2023-04-13 02:54:04,638 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/1188dd18aa6745e7b2c9c771adf8af3d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/1188dd18aa6745e7b2c9c771adf8af3d 2023-04-13 02:54:04,639 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/172b23f0524449e2ac37534fbd1d2006 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/172b23f0524449e2ac37534fbd1d2006 2023-04-13 02:54:04,641 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e540e00e38e4f3a8333868322c5d7de to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/9e540e00e38e4f3a8333868322c5d7de 2023-04-13 02:54:04,642 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aa7ab1fef961444eb98930c23e0f6d31 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/A/aa7ab1fef961444eb98930c23e0f6d31 2023-04-13 02:54:04,644 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/20c6c231aeb14bd6810480200a13d0cc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/20c6c231aeb14bd6810480200a13d0cc 2023-04-13 02:54:04,645 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/78c2cfe3f7704ae992a4282210ca01d6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/78c2cfe3f7704ae992a4282210ca01d6 2023-04-13 02:54:04,646 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/e53ef812af954ba2acd5544bd97530eb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/e53ef812af954ba2acd5544bd97530eb 2023-04-13 02:54:04,647 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f3c960f4eba945b2a9f6772215cdc809 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/B/f3c960f4eba945b2a9f6772215cdc809 2023-04-13 02:54:04,650 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2725befa80574134afde8f6561994d35 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/2725befa80574134afde8f6561994d35 2023-04-13 02:54:04,651 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/61a77699a6c049a5b8fb5a502a4673a9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/61a77699a6c049a5b8fb5a502a4673a9 2023-04-13 02:54:04,652 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d74c351eb9eb4f9885a52a9eb1f17e43 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/d74c351eb9eb4f9885a52a9eb1f17e43 2023-04-13 02:54:04,653 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e24533fd480f46998f851acb3a8d77df to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/C/e24533fd480f46998f851acb3a8d77df 2023-04-13 02:54:04,656 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/recovered.edits/1044.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e/recovered.edits/1044.seqid 2023-04-13 02:54:04,657 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/23777bf909bb0929ddad8e9cb44ee33e 2023-04-13 02:54:04,657 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:54:04,660 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,690 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:54:04,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 02:54:04,733 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:54:04,734 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,734 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:54:04,734 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354444734"}]},"ts":"9223372036854775807"} 2023-04-13 02:54:04,736 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:54:04,737 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 23777bf909bb0929ddad8e9cb44ee33e, NAME => 'TestAcidGuarantees,,1681354420573.23777bf909bb0929ddad8e9cb44ee33e.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:54:04,737 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:54:04,737 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354444737"}]},"ts":"9223372036854775807"} 2023-04-13 02:54:04,739 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:54:04,740 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,741 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 122 msec 2023-04-13 02:54:04,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 02:54:04,929 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-13 02:54:04,947 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_105728472_15 at /127.0.0.1:57182 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_105728472_15 at /127.0.0.1:59514 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cad558-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cad558-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cad558-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_105728472_15 at /127.0.0.1:59576 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x12cad558-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34919-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=579 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=463 (was 405) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=1060 (was 1705) 2023-04-13 02:54:04,958 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=579, MaxFileDescriptor=60000, SystemLoadAverage=463, ProcessCount=169, AvailableMemoryMB=1060 2023-04-13 02:54:04,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:54:04,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:54:04,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:04,962 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:54:04,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-13 02:54:04,963 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:54:04,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:04,965 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:04,965 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 empty. 2023-04-13 02:54:04,966 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:04,966 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:54:05,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:05,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:05,374 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:54:05,376 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fbff9ea8a72ff1f3ab1b323ef1e5cd25, NAME => 'TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:54:05,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fbff9ea8a72ff1f3ab1b323ef1e5cd25, disabling compactions & flushes 2023-04-13 02:54:05,784 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. after waiting 0 ms 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:05,784 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:05,784 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:05,787 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:54:05,788 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354445788"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354445788"}]},"ts":"1681354445788"} 2023-04-13 02:54:05,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:54:05,790 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:54:05,790 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354445790"}]},"ts":"1681354445790"} 2023-04-13 02:54:05,791 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:54:05,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, ASSIGN}] 2023-04-13 02:54:05,796 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, ASSIGN 2023-04-13 02:54:05,796 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:54:05,948 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:05,948 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354445947"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354445947"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354445947"}]},"ts":"1681354445947"} 2023-04-13 02:54:05,949 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:06,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:06,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:06,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fbff9ea8a72ff1f3ab1b323ef1e5cd25, NAME => 'TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:54:06,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:54:06,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,107 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,109 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A 2023-04-13 02:54:06,109 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A 2023-04-13 02:54:06,109 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:06,109 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName A 2023-04-13 02:54:06,110 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:06,110 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,111 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B 2023-04-13 02:54:06,111 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B 2023-04-13 02:54:06,111 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:06,111 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName B 2023-04-13 02:54:06,112 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:06,112 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,113 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C 2023-04-13 02:54:06,113 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C 2023-04-13 02:54:06,113 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:06,113 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName C 2023-04-13 02:54:06,114 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:06,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:06,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:54:06,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:06,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:54:06,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fbff9ea8a72ff1f3ab1b323ef1e5cd25; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66011608, jitterRate=-0.016350388526916504}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:54:06,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:06,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., pid=18, masterSystemTime=1681354446101 2023-04-13 02:54:06,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:06,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:06,123 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:06,123 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354446123"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354446123"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354446123"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354446123"}]},"ts":"1681354446123"} 2023-04-13 02:54:06,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-13 02:54:06,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 in 175 msec 2023-04-13 02:54:06,127 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-13 02:54:06,127 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, ASSIGN in 332 msec 2023-04-13 02:54:06,128 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:54:06,128 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354446128"}]},"ts":"1681354446128"} 2023-04-13 02:54:06,129 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:54:06,131 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:54:06,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-13 02:54:07,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 02:54:07,069 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-13 02:54:07,070 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x39cddc3f to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:07,076 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2586ae9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:07,078 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:07,083 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:07,085 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:54:07,088 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:54:07,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:54:07,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:54:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:07,519 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:54:07,521 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:54:07,521 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:54:07,526 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 02:54:07,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, REOPEN/MOVE}] 2023-04-13 02:54:07,540 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, REOPEN/MOVE 2023-04-13 02:54:07,541 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:07,541 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354447541"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354447541"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354447541"}]},"ts":"1681354447541"} 2023-04-13 02:54:07,543 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:07,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:07,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fbff9ea8a72ff1f3ab1b323ef1e5cd25, disabling compactions & flushes 2023-04-13 02:54:07,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:07,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:07,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. after waiting 0 ms 2023-04-13 02:54:07,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:07,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 02:54:07,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:07,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:07,701 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fbff9ea8a72ff1f3ab1b323ef1e5cd25 to self. 2023-04-13 02:54:07,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:07,703 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=CLOSED 2023-04-13 02:54:07,703 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354447703"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354447703"}]},"ts":"1681354447703"} 2023-04-13 02:54:07,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-13 02:54:07,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 in 161 msec 2023-04-13 02:54:07,707 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=true 2023-04-13 02:54:07,857 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:07,857 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354447857"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354447857"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354447857"}]},"ts":"1681354447857"} 2023-04-13 02:54:07,859 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:08,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fbff9ea8a72ff1f3ab1b323ef1e5cd25, NAME => 'TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:54:08,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:54:08,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,018 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,019 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A 2023-04-13 02:54:08,019 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A 2023-04-13 02:54:08,019 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:08,022 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName A 2023-04-13 02:54:08,025 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:08,025 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,027 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B 2023-04-13 02:54:08,027 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B 2023-04-13 02:54:08,027 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:08,027 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName B 2023-04-13 02:54:08,028 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:08,028 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,029 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C 2023-04-13 02:54:08,029 DEBUG [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C 2023-04-13 02:54:08,029 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:08,030 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fbff9ea8a72ff1f3ab1b323ef1e5cd25 columnFamilyName C 2023-04-13 02:54:08,031 INFO [StoreOpener-fbff9ea8a72ff1f3ab1b323ef1e5cd25-1] regionserver.HStore(310): Store=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:08,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:54:08,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fbff9ea8a72ff1f3ab1b323ef1e5cd25; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72466734, jitterRate=0.07983848452568054}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:54:08,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:08,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., pid=23, masterSystemTime=1681354448011 2023-04-13 02:54:08,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,040 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,041 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354448040"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354448040"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354448040"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354448040"}]},"ts":"1681354448040"} 2023-04-13 02:54:08,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-13 02:54:08,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 in 183 msec 2023-04-13 02:54:08,046 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-13 02:54:08,046 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, REOPEN/MOVE in 505 msec 2023-04-13 02:54:08,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-13 02:54:08,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 523 msec 2023-04-13 02:54:08,052 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 953 msec 2023-04-13 02:54:08,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-13 02:54:08,063 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x695af552 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,075 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7adf5e8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,076 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x550c7a27 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,080 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18496e17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,081 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x40c57efc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,088 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76aef981, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,088 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x4183389c to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,094 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40e0a3dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,095 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x3bccfdc5 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,105 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63d375f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,106 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1345b761 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,112 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b416c44, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,112 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1e0b0a9f to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,118 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@204b5880, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,118 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x078e8770 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,124 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41173aec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,124 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x3a77cdd0 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:08,135 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@493168e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:08,141 DEBUG [hconnection-0x12051855-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,142 DEBUG [hconnection-0x5d7da072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,144 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:08,148 DEBUG [hconnection-0x4fe6e4ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,152 DEBUG [hconnection-0x32f0b4b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,153 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,159 DEBUG [hconnection-0x141ee72d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,160 DEBUG [hconnection-0x5cdb4c18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,164 DEBUG [hconnection-0x5b9818f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,164 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,170 DEBUG [hconnection-0x78420db7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,170 DEBUG [hconnection-0x4390ee54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:08,171 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,171 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,171 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,172 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,173 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,177 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:08,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:08,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:08,184 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:08,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:08,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:08,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,192 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:08,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:08,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:54:08,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:08,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:08,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:08,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:08,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:08,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:08,217 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354508231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354508234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354508234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:08,241 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:08,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354508234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354508236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,243 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,243 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:08,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:08,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2cd6f5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:08,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:08,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:08,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:08,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:08,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:08,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,251 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:08,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:08,254 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:08,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:08,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:08,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:08,254 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:08,256 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,256 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,256 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,258 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:08,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,261 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:08,262 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:08,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354508338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354508341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354508342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354508343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354508343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,362 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:08,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:08,400 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:08,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:08,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:08,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,402 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,402 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:08,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:08,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:08,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:08,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:08,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:08,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:08,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:08,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:08,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:08,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:08,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:08,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:08,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:08,443 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,443 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,444 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f0efc35[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:08,445 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:08,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,447 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,447 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:08,447 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:08,449 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:08,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:08,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:08,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:08,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:08,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:08,453 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:08,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,456 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:08,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:08,464 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:08,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:08,464 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,464 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:08,465 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,465 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,465 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354508540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354508544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354508545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354508547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354508547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,564 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:08,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:08,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:08,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:08,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:08,612 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:08,613 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,613 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:08,613 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:08,613 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,613 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,616 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:08,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:08,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:08,626 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:08,636 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:08,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1553f3cd[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:08,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:08,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,639 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:08,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:08,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:08,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:08,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:08,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:08,642 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:08,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:08,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:08,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:08,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:08,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:08,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:08,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:08,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:08,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,657 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:08,657 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:08,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,690 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041319242816840a444597f00c570238930d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319242816840a444597f00c570238930d 2023-04-13 02:54:08,757 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:08,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:08,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:08,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:08,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:08,776 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:08,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:08,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:08,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,779 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:08,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:08,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:08,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:08,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:08,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:08,784 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,784 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:08,784 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:08,784 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:08,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:08,786 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,786 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:08,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b038322[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:08,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:08,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:08,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:08,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:08,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:08,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:08,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:08,800 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:08,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:08,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:08,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:08,800 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:08,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:08,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:08,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,809 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:08,809 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:08,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354508843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354508847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354508848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354508854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:08,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354508855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,909 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:08,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:08,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:08,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:08,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:08,915 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:08,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:08,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:08,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:08,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:08,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272465c9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:08,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:08,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:08,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,927 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:08,930 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:08,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:08,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:08,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:08,930 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:08,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:08,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:08,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:08,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:08,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:08,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:08,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:08,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:08,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:08,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:08,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:08,938 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:08,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:08,938 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:08,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:08,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,038 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,043 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,062 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f83ca4c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,081 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,081 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,093 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,101 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,101 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,101 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,101 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,112 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,112 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:09,120 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/bc68a20a2f5b4ea68c955d37b3945f5e 2023-04-13 02:54:09,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/001042ea6e3b4bc8891e971355d1e9a8 2023-04-13 02:54:09,212 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,222 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,231 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,232 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,232 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/3b3c67c5d1f34c689787759a263fb0e1 2023-04-13 02:54:09,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/bc68a20a2f5b4ea68c955d37b3945f5e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e 2023-04-13 02:54:09,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e, entries=150, sequenceid=17, filesize=25.1 K 2023-04-13 02:54:09,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/001042ea6e3b4bc8891e971355d1e9a8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8 2023-04-13 02:54:09,252 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,252 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 02:54:09,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/3b3c67c5d1f34c689787759a263fb0e1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1 2023-04-13 02:54:09,260 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,260 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,260 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,260 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 02:54:09,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1059ms, sequenceid=17, compaction requested=false 2023-04-13 02:54:09,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:09,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654481ea[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,285 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,285 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,285 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,285 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,291 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,291 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,303 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,304 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,304 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:09,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:09,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354509374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354509374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354509375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354509376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354509380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,401 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041344a1c848bcb746e2aec1e64d894e1486 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344a1c848bcb746e2aec1e64d894e1486 2023-04-13 02:54:09,404 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,427 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,433 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,433 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,436 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,436 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625914f3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,442 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,446 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,458 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,458 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,463 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,464 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,473 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,473 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:09,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354509482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354509482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354509483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354509484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354509486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,573 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,579 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,584 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,584 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,584 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,584 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,586 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,586 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,586 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d72f5c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,589 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,589 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,590 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,591 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,591 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,594 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,598 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,598 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,603 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,603 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,603 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,606 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,612 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,612 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,612 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354509685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354509685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354509686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354509689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354509691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,713 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,717 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,722 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,722 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,722 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,722 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f75860[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,728 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,731 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,732 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,744 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,745 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,745 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,831 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/3d31041a971d4683aa0941bf14401688 2023-04-13 02:54:09,844 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:54:09,845 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:09,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:09,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:09,850 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:09,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:09,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:09,852 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,852 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,853 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,855 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,855 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:09,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:09,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:09,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:09,867 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:09,867 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:09,867 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,868 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:09,868 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:09,868 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:09,871 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:09,871 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:09,871 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/b978343aefef44eead1771e122522823 2023-04-13 02:54:09,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:09,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:09,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,886 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:09,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567dd0a5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:09,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:09,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:09,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:09,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:09,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:09,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:09,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:09,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:09,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:09,912 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:09,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:09,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:09,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:09,912 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:09,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:09,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:09,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:09,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:09,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:09,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:09,920 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:09,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:09,920 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:09,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:09,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/73873a9447834a8f8c1367ed4d90dd04 2023-04-13 02:54:09,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/3d31041a971d4683aa0941bf14401688 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688 2023-04-13 02:54:09,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688, entries=200, sequenceid=37, filesize=31.9 K 2023-04-13 02:54:09,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/b978343aefef44eead1771e122522823 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823 2023-04-13 02:54:09,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823, entries=150, sequenceid=37, filesize=11.6 K 2023-04-13 02:54:09,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/73873a9447834a8f8c1367ed4d90dd04 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04 2023-04-13 02:54:09,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04, entries=150, sequenceid=37, filesize=11.6 K 2023-04-13 02:54:09,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 627ms, sequenceid=37, compaction requested=false 2023-04-13 02:54:09,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:09,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:09,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:09,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:09,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:09,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:09,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:09,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:09,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:10,020 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,031 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354510024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354510026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354510029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,035 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354510033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354510033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca9a815[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,062 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,065 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,065 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,074 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,074 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:10,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354510135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354510135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354510136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354510139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354510139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,174 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,178 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,181 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,181 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,181 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,182 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,182 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,182 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,182 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dbc513[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,195 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,196 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,206 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,207 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,207 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,307 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,312 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,320 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,321 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,321 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec2bd50[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,326 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,329 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,329 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,330 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,333 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,335 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,338 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,339 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,339 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354510337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354510338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354510339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354510342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354510344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,439 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,443 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,446 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,456 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3682acd5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,461 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,464 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,464 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,464 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,464 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,464 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,466 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304139b43a8b48ee64e6f958602f7e5145bbb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139b43a8b48ee64e6f958602f7e5145bbb 2023-04-13 02:54:10,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,467 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,467 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,472 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/fba4a3a7b8744e1ebba800a90adc6b35 2023-04-13 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,488 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,496 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,496 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,496 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,496 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,596 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,600 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,602 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,602 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,604 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,604 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,604 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,604 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,605 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,605 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,608 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,608 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a4310[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,610 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,611 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,612 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,615 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,615 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,619 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,624 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,624 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,624 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:10,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354510641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354510641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354510642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354510644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:10,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354510647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,724 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,732 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,738 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:10,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,740 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,740 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,741 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,741 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,742 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,742 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad4edb5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,745 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,745 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,747 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,747 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,747 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,756 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,760 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,760 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,771 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,771 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:10,873 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:10,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:10,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:10,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:10,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:10,880 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:10,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:10,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:10,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,882 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,882 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,882 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,882 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:10,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:10,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:10,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:10,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:10,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:10,886 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:10,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6167f782[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:10,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:10,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,889 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:10,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:10,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:10,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:10,891 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:10,891 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,891 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:10,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:10,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:10,895 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:10,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:10,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:10,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:10,895 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:10,896 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,896 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,896 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/ee3229162db646f1b59a0f418e2f3467 2023-04-13 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:10,901 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:10,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:10,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:10,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:10,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:10,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:10,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:10,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:10,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:10,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:10,908 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:10,908 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:10,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:10,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:11,008 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:11,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:11,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:11,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:11,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:11,013 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:11,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:11,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:11,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,015 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,015 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:11,017 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:11,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:11,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:11,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:11,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:11,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:11,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:11,020 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,021 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:11,021 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:11,021 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:11,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:11,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:11,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:11,026 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:11,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a00b7f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:11,028 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:11,028 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:11,030 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:11,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:11,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:11,033 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:11,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:11,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:11,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:11,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:11,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:11,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:11,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:11,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,048 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:11,048 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:11,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:11,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,149 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:11,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:11,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:11,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354511146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:11,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354511146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354511147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354511149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:11,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:11,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:11,154 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:11,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:11,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:11,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:11,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354511153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,157 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,158 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,158 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:11,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:11,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:11,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:11,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:11,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:11,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:11,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:11,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:11,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:11,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:11,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173792f8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:11,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:11,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:11,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:11,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:11,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:11,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:11,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:11,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:11,177 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:11,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:11,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:11,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:11,177 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:11,179 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,179 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,181 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:11,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:11,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,185 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:11,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,185 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:11,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,285 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:11,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:11,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:11,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:11,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:11,291 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:11,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:11,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:11,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:11,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:11,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:11,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:11,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:11,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:11,309 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,309 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:11,309 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:11,309 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:11,310 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:11,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:11,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a536cee[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:11,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:11,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:11,318 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:11,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:11,321 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:11,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:11,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:11,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:11,321 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:11,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/80fe03638d984675855f3840af9a4948 2023-04-13 02:54:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:11,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:11,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:11,336 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:11,336 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:11,335 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:11,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:11,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/fba4a3a7b8744e1ebba800a90adc6b35 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35 2023-04-13 02:54:11,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:11,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:11,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:11,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35, entries=150, sequenceid=51, filesize=25.1 K 2023-04-13 02:54:11,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/ee3229162db646f1b59a0f418e2f3467 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467 2023-04-13 02:54:11,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 02:54:11,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/80fe03638d984675855f3840af9a4948 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948 2023-04-13 02:54:11,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 02:54:11,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1397ms, sequenceid=51, compaction requested=true 2023-04-13 02:54:11,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:11,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:11,389 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:11,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:11,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:54:11,389 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:11,391 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:11,391 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/A is initiating minor compaction (all files) 2023-04-13 02:54:11,391 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/A in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,391 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=82.2 K 2023-04-13 02:54:11,393 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:11,393 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bc68a20a2f5b4ea68c955d37b3945f5e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681354448196 2023-04-13 02:54:11,393 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/B is initiating minor compaction (all files) 2023-04-13 02:54:11,393 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/B in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,393 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=34.8 K 2023-04-13 02:54:11,393 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3d31041a971d4683aa0941bf14401688, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681354448229 2023-04-13 02:54:11,394 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting fba4a3a7b8744e1ebba800a90adc6b35, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:11,394 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 001042ea6e3b4bc8891e971355d1e9a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681354448196 2023-04-13 02:54:11,395 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b978343aefef44eead1771e122522823, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681354448229 2023-04-13 02:54:11,397 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ee3229162db646f1b59a0f418e2f3467, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:11,431 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#B#compaction#121 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:11,436 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:11,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:11,445 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#A#compaction#120 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:11,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:11,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:11,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:11,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:11,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:11,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:11,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:11,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:11,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:11,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:11,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:11,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:11,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:11,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:11,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:11,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:11,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:11,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:11,463 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 02:54:11,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:11,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:11,464 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:11,464 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:11,464 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:11,464 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:11,489 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/21f14a81261340f4a0c4d3cc946c88a1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/21f14a81261340f4a0c4d3cc946c88a1 2023-04-13 02:54:11,497 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/B of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 21f14a81261340f4a0c4d3cc946c88a1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:11,497 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:11,498 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, priority=13, startTime=1681354451389; duration=0sec 2023-04-13 02:54:11,498 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:11,498 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:11,500 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:11,500 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/C is initiating minor compaction (all files) 2023-04-13 02:54:11,500 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/C in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:11,500 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=34.8 K 2023-04-13 02:54:11,501 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3b3c67c5d1f34c689787759a263fb0e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681354448196 2023-04-13 02:54:11,501 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 73873a9447834a8f8c1367ed4d90dd04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681354448229 2023-04-13 02:54:11,502 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 80fe03638d984675855f3840af9a4948, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:11,517 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#C#compaction#123 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:11,523 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ac0bb3bbf1c24a9682caac5514c90e4e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ac0bb3bbf1c24a9682caac5514c90e4e 2023-04-13 02:54:11,562 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/e18a8be28cc14286b0e62a04c405557a 2023-04-13 02:54:11,570 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/3dd6a9de409e4e7ea47dbb7dae707d03 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3dd6a9de409e4e7ea47dbb7dae707d03 2023-04-13 02:54:11,583 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/C of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 3dd6a9de409e4e7ea47dbb7dae707d03(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:11,583 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:11,583 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, priority=13, startTime=1681354451389; duration=0sec 2023-04-13 02:54:11,583 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:12,004 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/40cfbee042454df3a5979dcea2d1e49a 2023-04-13 02:54:12,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:12,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354512160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354512161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354512164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354512165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354512165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354512266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354512267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354512268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354512269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354512269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,337 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/f791937b4ea94e50812e2bbf79e38df3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f791937b4ea94e50812e2bbf79e38df3 2023-04-13 02:54:12,347 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/A of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into f791937b4ea94e50812e2bbf79e38df3(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:12,347 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:12,347 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, priority=13, startTime=1681354451389; duration=0sec 2023-04-13 02:54:12,348 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:12,419 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/106e726f77cc4e4f893ee5dbadfbc2e5 2023-04-13 02:54:12,426 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/e18a8be28cc14286b0e62a04c405557a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a 2023-04-13 02:54:12,434 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a, entries=150, sequenceid=70, filesize=25.1 K 2023-04-13 02:54:12,435 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/40cfbee042454df3a5979dcea2d1e49a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a 2023-04-13 02:54:12,444 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a, entries=150, sequenceid=70, filesize=11.6 K 2023-04-13 02:54:12,446 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/106e726f77cc4e4f893ee5dbadfbc2e5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5 2023-04-13 02:54:12,451 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5, entries=150, sequenceid=70, filesize=11.6 K 2023-04-13 02:54:12,452 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 989ms, sequenceid=70, compaction requested=false 2023-04-13 02:54:12,453 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:12,453 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:12,457 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145649c1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:12,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:12,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:12,461 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,461 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:12,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:12,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:12,472 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:12,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:12,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:12,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:12,472 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:12,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:54:12,477 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,477 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,477 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,478 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:12,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,480 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,483 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,483 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,483 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,483 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:12,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:12,483 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,484 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:12,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:12,484 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:12,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354512488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354512490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354512492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354512492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354512494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,584 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:12,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:12,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:12,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:12,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:12,592 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:12,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:12,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:12,593 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,593 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,595 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,596 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354512595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,597 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,597 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:12,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:12,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:12,600 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:12,601 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:12,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:12,602 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354512599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:12,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566f43c7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:12,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:12,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354512599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354512599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354512600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,606 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:12,606 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:12,606 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:12,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:12,610 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:12,617 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:12,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:12,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:12,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:12,617 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:12,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,625 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:12,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,628 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:12,628 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:12,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,728 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:12,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:12,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:12,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:12,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:12,736 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:12,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:12,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:12,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,738 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,738 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,738 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,738 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:12,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:12,744 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,744 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:12,744 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:12,744 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:12,746 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,746 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,746 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,746 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d64aba0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:12,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:12,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,751 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:12,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:12,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:12,756 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:12,756 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:12,756 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:12,756 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:12,756 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:12,758 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,758 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,758 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,761 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:12,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:12,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,765 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:12,765 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:12,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354512799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354512807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354512807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:12,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354512807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354512808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,865 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:12,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:12,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:12,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:12,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:12,869 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:12,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:12,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:12,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,872 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,873 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,874 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:12,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:12,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:12,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:12,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:12,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:12,878 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c018cfe[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:12,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:12,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:12,883 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,883 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:12,884 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:12,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:12,887 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:12,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:12,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:12,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:12,887 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:12,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:12,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:12,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:12,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:12,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:12,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:12,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:12,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:12,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:12,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:12,898 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:12,898 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:12,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:12,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:12,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:12,924 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041344c108d586c84f6f98b2f145fc8849ec to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344c108d586c84f6f98b2f145fc8849ec 2023-04-13 02:54:12,998 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:12,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,001 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,008 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,010 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,011 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,013 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abfd0fe[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,015 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,015 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,016 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,017 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,020 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,020 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,024 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,028 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,028 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354513105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354513113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354513115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354513116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354513117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,128 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,132 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,135 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,135 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,136 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,136 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,137 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,138 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,138 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2662cb15[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,145 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,147 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,151 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,151 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,151 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,160 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,160 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,160 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,260 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,265 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,266 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,267 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,270 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,270 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,270 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,270 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,272 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e035dd2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,276 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,290 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,293 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,293 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,295 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,303 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,304 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,304 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,335 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=91, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/dd6289030a014d2f899d7faa486ec83e 2023-04-13 02:54:13,404 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,408 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,417 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e454890[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,422 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,422 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,427 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,427 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,434 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,434 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,534 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,538 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,545 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,545 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,545 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a095b86[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,552 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,554 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,554 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,561 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,561 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,561 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354513612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354513619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354513620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354513621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354513624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,662 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,665 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0307b1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,674 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,679 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,679 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,689 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,690 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a23de86bb0de4830bc300b4e7e0f2e3f 2023-04-13 02:54:13,790 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,793 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8a24ee[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,810 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,810 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,819 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,819 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,919 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:13,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:13,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:13,923 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:13,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:13,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:13,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:13,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:13,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:13,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:13,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:13,929 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,929 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:13,929 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:13,929 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:13,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@affe267[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:13,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:13,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:13,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:13,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:13,942 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:13,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:13,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:13,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:13,942 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:13,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:13,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:13,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:13,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:13,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:13,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:13,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:13,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:13,953 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:13,953 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:13,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:13,954 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:13,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:14,053 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:14,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:14,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:14,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:14,056 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:14,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:14,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:14,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:14,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:14,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:14,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:14,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:14,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:14,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:14,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:14,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:14,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:14,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:14,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:14,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4a5dd5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:14,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:14,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:14,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:14,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:14,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:14,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:14,077 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:14,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:14,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:14,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:14,077 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:14,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:14,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:14,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:14,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,082 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:14,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:14,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:14,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:14,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:14,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,086 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:14,086 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:14,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:14,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:14,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e 2023-04-13 02:54:14,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/dd6289030a014d2f899d7faa486ec83e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e 2023-04-13 02:54:14,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e, entries=150, sequenceid=91, filesize=25.1 K 2023-04-13 02:54:14,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a23de86bb0de4830bc300b4e7e0f2e3f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f 2023-04-13 02:54:14,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f, entries=150, sequenceid=91, filesize=11.6 K 2023-04-13 02:54:14,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e 2023-04-13 02:54:14,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e, entries=150, sequenceid=91, filesize=11.6 K 2023-04-13 02:54:14,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1709ms, sequenceid=91, compaction requested=true 2023-04-13 02:54:14,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:14,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:14,185 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:14,185 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:14,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:14,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:14,186 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:14,186 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:14,187 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/B is initiating minor compaction (all files) 2023-04-13 02:54:14,187 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/B in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:14,187 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/21f14a81261340f4a0c4d3cc946c88a1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=34.9 K 2023-04-13 02:54:14,188 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:14,188 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/A is initiating minor compaction (all files) 2023-04-13 02:54:14,188 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/A in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,188 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f791937b4ea94e50812e2bbf79e38df3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=75.5 K 2023-04-13 02:54:14,189 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 21f14a81261340f4a0c4d3cc946c88a1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:14,189 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f791937b4ea94e50812e2bbf79e38df3, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:14,190 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 40cfbee042454df3a5979dcea2d1e49a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681354450023 2023-04-13 02:54:14,190 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a23de86bb0de4830bc300b4e7e0f2e3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:14,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:14,192 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e18a8be28cc14286b0e62a04c405557a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681354450023 2023-04-13 02:54:14,193 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dd6289030a014d2f899d7faa486ec83e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:14,193 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:14,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:14,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:14,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:14,194 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,194 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,196 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:14,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:14,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:14,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:14,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:14,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:14,212 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:14,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:14,213 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#B#compaction#130 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:14,226 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#A#compaction#129 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:14,248 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041304deaf653eb14681841b1c7809591cfa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041304deaf653eb14681841b1c7809591cfa 2023-04-13 02:54:14,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:14,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:14,654 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/ae834612e42b4a95a64beacd0999f178 2023-04-13 02:54:14,655 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/d74b4b75540d4b7ebba0306fe26e0a4f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/d74b4b75540d4b7ebba0306fe26e0a4f 2023-04-13 02:54:14,665 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/B of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into d74b4b75540d4b7ebba0306fe26e0a4f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:14,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:14,665 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, priority=13, startTime=1681354454185; duration=0sec 2023-04-13 02:54:14,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:14,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:14,666 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:14,666 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/C is initiating minor compaction (all files) 2023-04-13 02:54:14,666 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/C in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:14,667 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3dd6a9de409e4e7ea47dbb7dae707d03, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=34.9 K 2023-04-13 02:54:14,667 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3dd6a9de409e4e7ea47dbb7dae707d03, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354449378 2023-04-13 02:54:14,668 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 106e726f77cc4e4f893ee5dbadfbc2e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681354450023 2023-04-13 02:54:14,668 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1ac7efe4411c4f26aa9d0cbdf3e2dd8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:14,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354514666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354514666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354514668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354514669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354514670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,685 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#C#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:14,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354514772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354514772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354514772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354514772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354514773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354514973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354514974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354514974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354514974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354514975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,088 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/db8d35fd22a74f6c8835aa8e075b8149 2023-04-13 02:54:15,110 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/2f821ee4268c476f9257782e503a5e56 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/2f821ee4268c476f9257782e503a5e56 2023-04-13 02:54:15,122 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/C of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 2f821ee4268c476f9257782e503a5e56(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:15,122 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:15,122 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, priority=13, startTime=1681354454185; duration=0sec 2023-04-13 02:54:15,122 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:15,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354515278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354515279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354515279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354515279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354515280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,472 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/574fb196753a4a65aaab537a331da76e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/574fb196753a4a65aaab537a331da76e 2023-04-13 02:54:15,477 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/A of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 574fb196753a4a65aaab537a331da76e(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 02:54:15,477 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:15,477 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, priority=13, startTime=1681354454185; duration=1sec 2023-04-13 02:54:15,477 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:15,501 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/61448e0b13c749d9922a05ba98ad33d8 2023-04-13 02:54:15,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/ae834612e42b4a95a64beacd0999f178 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178 2023-04-13 02:54:15,511 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178, entries=150, sequenceid=105, filesize=25.1 K 2023-04-13 02:54:15,512 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/db8d35fd22a74f6c8835aa8e075b8149 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149 2023-04-13 02:54:15,517 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149, entries=150, sequenceid=105, filesize=11.6 K 2023-04-13 02:54:15,518 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/61448e0b13c749d9922a05ba98ad33d8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8 2023-04-13 02:54:15,522 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8, entries=150, sequenceid=105, filesize=11.6 K 2023-04-13 02:54:15,523 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1310ms, sequenceid=105, compaction requested=false 2023-04-13 02:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:15,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:15,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6876e1a2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:15,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:15,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,531 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:15,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:15,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:15,533 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:15,535 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:15,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:15,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:15,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:15,536 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:15,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:15,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:15,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:15,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:15,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:15,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:15,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,544 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:15,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,544 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,644 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:15,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:15,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:15,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:15,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:15,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:15,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:15,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:15,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:15,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,649 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:15,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:15,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:15,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:15,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:15,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:15,651 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:15,652 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:15,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,667 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ea1575b26d0a4fb6aa2ecfb6071635aa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ea1575b26d0a4fb6aa2ecfb6071635aa 2023-04-13 02:54:15,674 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/21853100f8564794b71c219081fe52e7 2023-04-13 02:54:15,686 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/1d6b98bf580f41678d0d71fcdd31dec2 2023-04-13 02:54:15,697 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/0409d2704ef54c5bb8458bd9e244d69b 2023-04-13 02:54:15,702 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/21853100f8564794b71c219081fe52e7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7 2023-04-13 02:54:15,707 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7, entries=150, sequenceid=126, filesize=25.1 K 2023-04-13 02:54:15,708 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/1d6b98bf580f41678d0d71fcdd31dec2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2 2023-04-13 02:54:15,712 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2, entries=150, sequenceid=126, filesize=11.6 K 2023-04-13 02:54:15,713 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/0409d2704ef54c5bb8458bd9e244d69b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b 2023-04-13 02:54:15,723 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b, entries=150, sequenceid=126, filesize=11.6 K 2023-04-13 02:54:15,724 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 72ms, sequenceid=126, compaction requested=true 2023-04-13 02:54:15,724 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:15,724 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:15,724 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:15,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,731 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f41516[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:15,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:15,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,738 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:15,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:15,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:15,740 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:15,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:15,743 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:15,743 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:15,743 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:15,743 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:15,743 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:15,748 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,748 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,748 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:15,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,754 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:15,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:15,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:15,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,755 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:15,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,755 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:15,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:15,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:15,855 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:15,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:15,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:15,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:15,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:15,864 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:15,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:15,865 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,865 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,867 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,867 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:15,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,868 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,868 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:15,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:15,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:15,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:15,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:15,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:15,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354515876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354515877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354515878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354515878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354455874, max timestamp=1681354455880Num uniques 859; 2023-04-13 02:54:15,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354455874, max timestamp=1681354455880Num uniques 859; 2023-04-13 02:54:15,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354455874, max timestamp=1681354455880Num uniques 859; 2023-04-13 02:54:15,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:15,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:15,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:15,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:15,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:15,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:15,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:15,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:15,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:15,894 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:15,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:15,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ed2b55[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:15,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:15,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,899 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:15,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:15,904 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:15,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:15,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:15,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:15,904 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:15,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:15,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,908 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:15,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:15,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:15,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:15,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:15,912 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:15,912 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:15,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:15,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354515939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354515983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354515983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354515983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:15,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:15,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354515985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,012 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,016 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,016 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,019 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,019 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,020 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,020 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,021 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a716c28[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,028 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,028 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,028 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,030 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,032 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,032 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,038 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,038 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354516042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,138 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,142 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,149 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d5f735[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,156 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,157 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,160 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,160 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,167 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,167 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354516185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354516186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354516186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354516187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354516244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,267 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,270 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,272 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,276 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326e043c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,280 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,283 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137da71b9ad9e44d6ca9df347dac6e10ea to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137da71b9ad9e44d6ca9df347dac6e10ea 2023-04-13 02:54:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,286 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,286 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,290 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,293 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,293 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,394 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,397 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,400 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,403 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690226a7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,406 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,406 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,406 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,408 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,410 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,411 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,421 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,421 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354516487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354516490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354516491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354516491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,521 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,525 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68131783[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354516547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,549 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,549 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,649 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,653 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,660 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128529eb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,666 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,667 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,680 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,680 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,680 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,695 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/d475def9d9b84b21bca040511300a93f 2023-04-13 02:54:16,781 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,784 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,786 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,787 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,787 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50089063[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,802 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,820 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,820 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,920 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:16,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:16,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:16,923 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:16,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:16,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:16,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:16,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:16,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:16,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:16,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:16,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:16,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:16,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:16,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:16,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2847b3db[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:16,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:16,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:16,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:16,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:16,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:16,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:16,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:16,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:16,938 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:16,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:16,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:16,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:16,938 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:16,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:16,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,941 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:16,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:16,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:16,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:16,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:16,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:16,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:16,944 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:16,944 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:16,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:16,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:16,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:16,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:16,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354516993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354516995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354516996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:16,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:16,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354516996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,044 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:17,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:17,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:17,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:17,049 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:17,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:17,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:17,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:17,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,051 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:17,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:17,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:17,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:17,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354517053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:17,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:17,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:17,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:17,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:17,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5818165f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:17,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:17,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,070 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:17,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:17,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:17,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:17,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:17,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:17,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:17,076 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:17,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:17,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:17,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:17,076 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:17,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:17,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:17,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:17,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:17,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,089 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:17,089 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,089 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,089 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:17,089 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:17,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:17,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:17,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:17,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:17,090 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:17,090 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:17,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:17,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/65176bff082c4c5086e84921c5fcc3af 2023-04-13 02:54:17,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/d55282830858419ca503c58c17e7f8c2 2023-04-13 02:54:17,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/d475def9d9b84b21bca040511300a93f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f 2023-04-13 02:54:17,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f, entries=400, sequenceid=137, filesize=59.4 K 2023-04-13 02:54:17,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/65176bff082c4c5086e84921c5fcc3af as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af 2023-04-13 02:54:17,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af, entries=150, sequenceid=137, filesize=11.8 K 2023-04-13 02:54:17,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/d55282830858419ca503c58c17e7f8c2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2 2023-04-13 02:54:17,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2, entries=150, sequenceid=137, filesize=11.8 K 2023-04-13 02:54:17,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1339ms, sequenceid=137, compaction requested=true 2023-04-13 02:54:17,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:17,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:17,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:17,182 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:17,182 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:17,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:17,183 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:17,183 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138298 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:17,183 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/B is initiating minor compaction (all files) 2023-04-13 02:54:17,183 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/A is initiating minor compaction (all files) 2023-04-13 02:54:17,183 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/B in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,183 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/A in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,184 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/d74b4b75540d4b7ebba0306fe26e0a4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=46.8 K 2023-04-13 02:54:17,184 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/574fb196753a4a65aaab537a331da76e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=135.1 K 2023-04-13 02:54:17,184 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d74b4b75540d4b7ebba0306fe26e0a4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:17,184 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 574fb196753a4a65aaab537a331da76e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:17,184 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting db8d35fd22a74f6c8835aa8e075b8149, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681354452487 2023-04-13 02:54:17,185 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ae834612e42b4a95a64beacd0999f178, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681354452487 2023-04-13 02:54:17,185 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1d6b98bf580f41678d0d71fcdd31dec2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354454655 2023-04-13 02:54:17,185 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 21853100f8564794b71c219081fe52e7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354454655 2023-04-13 02:54:17,185 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 65176bff082c4c5086e84921c5fcc3af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455837 2023-04-13 02:54:17,186 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d475def9d9b84b21bca040511300a93f, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455834 2023-04-13 02:54:17,191 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:17,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:17,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:17,202 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#B#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:17,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:17,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:17,216 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:17,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:17,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:17,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:17,217 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,217 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:17,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:17,219 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,219 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:17,219 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:17,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:17,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:17,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:17,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:17,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:17,236 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,236 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:17,236 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:17,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:17,250 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#A#compaction#142 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:17,261 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/9efaf7b74da446c0a61f4842ce6b915a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/9efaf7b74da446c0a61f4842ce6b915a 2023-04-13 02:54:17,274 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/B of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 9efaf7b74da446c0a61f4842ce6b915a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:17,274 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:17,274 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, priority=12, startTime=1681354457182; duration=0sec 2023-04-13 02:54:17,274 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:17,274 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:17,276 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:17,276 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/C is initiating minor compaction (all files) 2023-04-13 02:54:17,276 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/C in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:17,276 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/2f821ee4268c476f9257782e503a5e56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=46.8 K 2023-04-13 02:54:17,277 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2f821ee4268c476f9257782e503a5e56, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1681354452164 2023-04-13 02:54:17,277 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 61448e0b13c749d9922a05ba98ad33d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681354452487 2023-04-13 02:54:17,278 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0409d2704ef54c5bb8458bd9e244d69b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354454655 2023-04-13 02:54:17,279 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d55282830858419ca503c58c17e7f8c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455837 2023-04-13 02:54:17,293 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#C#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:17,311 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/5abffb766b5a491a9b692f9d728ee626 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/5abffb766b5a491a9b692f9d728ee626 2023-04-13 02:54:17,312 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/45b5642850a8409baee34609c1b041b1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/45b5642850a8409baee34609c1b041b1 2023-04-13 02:54:17,317 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/C of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 5abffb766b5a491a9b692f9d728ee626(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:17,317 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:17,317 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, priority=12, startTime=1681354457182; duration=0sec 2023-04-13 02:54:17,317 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:17,319 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/A of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 45b5642850a8409baee34609c1b041b1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:17,319 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:17,319 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, priority=12, startTime=1681354457182; duration=0sec 2023-04-13 02:54:17,320 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:17,684 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041331d3430c53954cfda60ee272326ee7ab to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041331d3430c53954cfda60ee272326ee7ab 2023-04-13 02:54:18,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:18,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:18,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354518008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354518009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354518009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354518012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354518065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,090 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=173, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/8dde9c852c0e45babde5718199f6d84e 2023-04-13 02:54:18,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354518114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354518114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354518115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354518115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354518317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354518319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354518319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354518319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,501 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/cfa1ae8724b54421b67283466132f03b 2023-04-13 02:54:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354518621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354518622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354518622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:18,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354518623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,916 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/aeb8c58fb13f47a7813af9bcd13a0341 2023-04-13 02:54:18,924 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/8dde9c852c0e45babde5718199f6d84e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e 2023-04-13 02:54:18,929 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e, entries=150, sequenceid=173, filesize=25.3 K 2023-04-13 02:54:18,930 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/cfa1ae8724b54421b67283466132f03b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b 2023-04-13 02:54:18,934 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b, entries=150, sequenceid=173, filesize=11.8 K 2023-04-13 02:54:18,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/aeb8c58fb13f47a7813af9bcd13a0341 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341 2023-04-13 02:54:18,941 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341, entries=150, sequenceid=173, filesize=11.8 K 2023-04-13 02:54:18,942 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1706ms, sequenceid=173, compaction requested=false 2023-04-13 02:54:18,942 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:18,942 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:18,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:18,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:18,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:18,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6369b6c6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:18,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:18,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:18,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:18,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:18,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:18,953 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:18,953 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:18,953 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:18,953 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:18,953 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:18,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:18,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:18,957 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:18,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:18,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:18,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:18,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:18,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:18,975 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,975 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:18,975 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:18,975 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:18,975 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:18,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:19,075 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:19,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:19,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:19,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:19,078 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:19,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:19,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:19,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:19,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:19,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:19,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:19,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:19,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:19,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:19,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:19,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:19,082 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:19,083 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:19,083 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:19,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:19,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:19,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354519141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354459135, max timestamp=1681354459142Num uniques 859; 2023-04-13 02:54:19,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354459135, max timestamp=1681354459142Num uniques 859; 2023-04-13 02:54:19,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:19,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354459135, max timestamp=1681354459142Num uniques 859; 2023-04-13 02:54:19,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354519142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:19,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354519180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354519182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354519245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354519246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354519283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354519283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354519448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354519448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354519486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354519486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,499 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304130f497eee8b564d65b59dd57bf0ec5803 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130f497eee8b564d65b59dd57bf0ec5803 2023-04-13 02:54:19,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354519751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354519753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354519787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:19,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354519788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:19,905 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/81192d2751ae4f3f8fa5bb3581961d15 2023-04-13 02:54:20,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:20,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354520088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,090 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:54:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354520256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:20,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354520256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354520293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:20,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354520296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,321 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/6df4e5e394fb44f28514a47228351fba 2023-04-13 02:54:20,737 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/702f49a5c0a348fcaf5c05fea1dabd50 2023-04-13 02:54:20,741 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/81192d2751ae4f3f8fa5bb3581961d15 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15 2023-04-13 02:54:20,745 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15, entries=150, sequenceid=187, filesize=25.3 K 2023-04-13 02:54:20,746 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/6df4e5e394fb44f28514a47228351fba as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba 2023-04-13 02:54:20,750 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 02:54:20,751 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/702f49a5c0a348fcaf5c05fea1dabd50 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50 2023-04-13 02:54:20,755 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 02:54:20,756 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1674ms, sequenceid=187, compaction requested=true 2023-04-13 02:54:20,756 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:20,756 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:20,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:20,758 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:20,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:20,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f6b940[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:20,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:20,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:20,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:20,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:20,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:20,765 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:20,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:20,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:20,768 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:20,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:20,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:20,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:20,768 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:20,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:20,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:20,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:20,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:20,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:20,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:20,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:20,774 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:20,774 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:20,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:20,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,874 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:20,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:20,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:20,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:20,878 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:20,880 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:20,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:20,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:20,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:20,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:20,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:20,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:20,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:20,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:20,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:20,883 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:20,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:20,897 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137f7f6b893ddf4eb8a6c2d2098a8684b5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137f7f6b893ddf4eb8a6c2d2098a8684b5 2023-04-13 02:54:20,909 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=222, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/3a351c0487b5404088ef7f1331b92bd4 2023-04-13 02:54:21,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:21,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:21,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354521276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354521277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354521302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354521305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,322 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/c69ec8fb8201481ea20f274de2837d6d 2023-04-13 02:54:21,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354521382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354521382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354521584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39670 deadline: 1681354521585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,733 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/b9f8ae104e8744d4ad9817bdd8289c37 2023-04-13 02:54:21,738 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/3a351c0487b5404088ef7f1331b92bd4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4 2023-04-13 02:54:21,743 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4, entries=150, sequenceid=222, filesize=25.3 K 2023-04-13 02:54:21,745 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/c69ec8fb8201481ea20f274de2837d6d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d 2023-04-13 02:54:21,751 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d, entries=150, sequenceid=222, filesize=11.8 K 2023-04-13 02:54:21,752 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/b9f8ae104e8744d4ad9817bdd8289c37 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37 2023-04-13 02:54:21,757 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37, entries=150, sequenceid=222, filesize=11.8 K 2023-04-13 02:54:21,758 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 875ms, sequenceid=222, compaction requested=true 2023-04-13 02:54:21,758 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:21,758 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:21,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:21,760 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:21,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623a050e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:21,761 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:21,761 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:21,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:21,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:21,766 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:21,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:21,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:21,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:21,767 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:21,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:21,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:21,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:21,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:21,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:21,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:21,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:21,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:21,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,774 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:21,774 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:21,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:21,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:21,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,874 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:21,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:21,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:21,878 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:21,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:21,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:21,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:21,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:21,880 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:21,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:21,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:21,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:21,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:21,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:21,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:21,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:21,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:21,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:21,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:21,882 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:54:21,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:21,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:21,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:21,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:21,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:21,883 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:21,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:21,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:21,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354521921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354461899, max timestamp=1681354461923Num uniques 859; 2023-04-13 02:54:21,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:54:21,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354461899, max timestamp=1681354461923Num uniques 859; 2023-04-13 02:54:21,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354461899, max timestamp=1681354461923Num uniques 859; 2023-04-13 02:54:21,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:54:21,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:54:21,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354521978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354522025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354522081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39710 deadline: 1681354522227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354522282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,302 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413674797a35dbd4285972d7a4570530a1e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413674797a35dbd4285972d7a4570530a1e 2023-04-13 02:54:22,307 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/c82b591d25454f5c90ab729e5fd0747d 2023-04-13 02:54:22,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354522530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:22,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354522588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,718 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a514d5bf5d37470cb600871a1d7450c2 2023-04-13 02:54:22,729 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/44ad14c923034dd4a1d05b01ca866c11 2023-04-13 02:54:22,735 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/c82b591d25454f5c90ab729e5fd0747d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d 2023-04-13 02:54:22,739 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d, entries=150, sequenceid=233, filesize=25.3 K 2023-04-13 02:54:22,740 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a514d5bf5d37470cb600871a1d7450c2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2 2023-04-13 02:54:22,745 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2, entries=150, sequenceid=233, filesize=11.8 K 2023-04-13 02:54:22,746 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/44ad14c923034dd4a1d05b01ca866c11 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11 2023-04-13 02:54:22,750 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11, entries=150, sequenceid=233, filesize=11.8 K 2023-04-13 02:54:22,751 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 869ms, sequenceid=233, compaction requested=true 2023-04-13 02:54:22,751 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:22,751 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:22,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:22,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:22,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a884df8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:22,755 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:22,755 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,757 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:22,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:22,757 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:22,757 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,757 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:22,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:22,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:22,758 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:22,758 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:22,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:22,762 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:22,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:22,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:22,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:22,762 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:22,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:22,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:22,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:22,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:22,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:22,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:22,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,767 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:22,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:22,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:22,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:22,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:22,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,768 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:22,768 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:22,868 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:22,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:22,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:22,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:22,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:22,872 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:22,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:22,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:22,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:22,873 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,873 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:22,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:22,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:22,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:22,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:22,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:22,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:22,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:22,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:22,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:22,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:22,877 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:22,877 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:22,891 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413fe05ba071b484adda1396dd98b57188d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fe05ba071b484adda1396dd98b57188d 2023-04-13 02:54:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:23,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:23,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354523055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354523093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354523156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,299 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=267, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/ef3d021b3d284fcf8c74e2bf7290d339 2023-04-13 02:54:23,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354523319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39642 deadline: 1681354523319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,322 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:54:23,323 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:54:23,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354523360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354523665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:23,713 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/48983f893cc24acf946d74caf553d0fe 2023-04-13 02:54:24,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354524097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,126 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/acfbfd65fd5048e1b7e711758290f46a 2023-04-13 02:54:24,131 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/ef3d021b3d284fcf8c74e2bf7290d339 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339 2023-04-13 02:54:24,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39714 deadline: 1681354524130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,135 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:54:24,136 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339, entries=150, sequenceid=267, filesize=25.4 K 2023-04-13 02:54:24,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/48983f893cc24acf946d74caf553d0fe as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe 2023-04-13 02:54:24,145 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe, entries=150, sequenceid=267, filesize=11.9 K 2023-04-13 02:54:24,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/acfbfd65fd5048e1b7e711758290f46a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a 2023-04-13 02:54:24,149 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a, entries=150, sequenceid=267, filesize=11.9 K 2023-04-13 02:54:24,150 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1273ms, sequenceid=267, compaction requested=true 2023-04-13 02:54:24,150 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:24,150 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,155 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54791be9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,157 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,158 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,158 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,162 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,162 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,165 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,165 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,165 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,170 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,170 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:24,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:54:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:24,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:24,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,171 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:24,171 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:24,184 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041373b354622fac4421b494bb246246e7b3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041373b354622fac4421b494bb246246e7b3 2023-04-13 02:54:24,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354524239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,270 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,273 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,275 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,277 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,277 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,277 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,277 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,279 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1687949f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,284 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,287 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,287 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,293 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,293 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354524341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,393 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,397 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,400 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,402 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2427e134[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,406 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,406 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,408 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,411 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,411 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,419 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,419 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,419 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,519 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,523 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,527 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf88fc5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,541 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,541 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,547 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,547 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354524546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,590 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/16bf91dd9fe34574b1d462035f01bb3e 2023-04-13 02:54:24,647 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,650 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,652 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,654 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,655 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c279e0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,658 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,663 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,663 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,665 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,665 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,665 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,670 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,670 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,770 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,773 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,775 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,777 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,777 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,777 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,777 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,780 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4b260b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,783 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,783 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,783 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,784 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,786 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,787 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,789 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,793 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,793 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354524849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,893 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:24,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:24,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:24,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:24,896 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:24,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:24,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:24,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,898 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:24,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:24,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:24,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:24,900 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,900 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:24,901 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:24,901 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:24,902 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e4c37[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:24,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:24,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:24,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:24,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:24,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:24,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:24,913 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:24,913 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:24,913 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:24,913 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:24,913 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:24,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:24,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:24,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:24,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:24,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:24,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:24,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:24,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:24,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:24,919 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:24,919 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:24,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:24,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:24,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:24,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:24,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/87e5348806584453b7fa767ed209edd6 2023-04-13 02:54:25,019 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:25,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:25,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:25,023 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:25,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:25,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:25,024 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,024 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,026 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:25,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:25,028 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,028 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:25,028 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:25,028 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:25,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:25,030 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,030 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69206b7a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:25,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:25,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:25,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:25,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:25,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:25,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:25,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:25,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:25,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:25,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:25,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:25,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:25,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:25,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,044 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:25,044 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:25,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,044 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,144 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:25,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:25,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:25,147 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:25,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:25,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:25,148 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,148 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,149 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,150 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,150 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:25,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:25,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:25,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:25,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:25,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:25,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7abe02c7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:25,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,157 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:25,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:25,158 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:25,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:25,161 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:25,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:25,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:25,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:25,161 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:25,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:25,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:25,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,167 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:25,167 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:25,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,267 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:25,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:25,270 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:25,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:25,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:25,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,273 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:25,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:25,275 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,275 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:25,275 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:25,275 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:25,277 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b608e5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:25,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:25,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,280 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:25,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:25,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:25,283 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:25,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:25,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:25,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:25,283 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:25,286 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,286 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,286 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:25,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,294 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:25,294 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:25,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:25,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354525352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,394 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:25,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:25,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:25,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:25,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:25,412 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:25,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:25,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:25,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:25,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:25,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:25,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:25,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:25,416 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:25,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/fc0fbabbcb23411fab3fe9e3c4e9aee1 2023-04-13 02:54:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:25,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/16bf91dd9fe34574b1d462035f01bb3e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a89606[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:25,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:25,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e, entries=200, sequenceid=279, filesize=32.3 K 2023-04-13 02:54:25,428 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-13 02:54:25,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/87e5348806584453b7fa767ed209edd6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6 2023-04-13 02:54:25,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:25,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6, entries=150, sequenceid=279, filesize=11.9 K 2023-04-13 02:54:25,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/fc0fbabbcb23411fab3fe9e3c4e9aee1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1 2023-04-13 02:54:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:25,441 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:25,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:25,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:25,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:25,442 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:25,443 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,443 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:25,443 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1, entries=150, sequenceid=279, filesize=11.9 K 2023-04-13 02:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1275ms, sequenceid=279, compaction requested=true 2023-04-13 02:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:25,445 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:25,446 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:54:25,446 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:54:25,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:25,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:25,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,448 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:25,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:25,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:25,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:25,449 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84894 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:54:25,448 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188954 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:54:25,449 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/B is initiating minor compaction (all files) 2023-04-13 02:54:25,449 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,449 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/B in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,449 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:25,449 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/9efaf7b74da446c0a61f4842ce6b915a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=82.9 K 2023-04-13 02:54:25,449 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/A is initiating minor compaction (all files) 2023-04-13 02:54:25,449 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:25,449 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/A in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,449 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/45b5642850a8409baee34609c1b041b1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=184.5 K 2023-04-13 02:54:25,449 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9efaf7b74da446c0a61f4842ce6b915a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455837 2023-04-13 02:54:25,449 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 45b5642850a8409baee34609c1b041b1, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455837 2023-04-13 02:54:25,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cfa1ae8724b54421b67283466132f03b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1681354455923 2023-04-13 02:54:25,450 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8dde9c852c0e45babde5718199f6d84e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1681354455923 2023-04-13 02:54:25,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6df4e5e394fb44f28514a47228351fba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681354458006 2023-04-13 02:54:25,450 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 81192d2751ae4f3f8fa5bb3581961d15, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681354458006 2023-04-13 02:54:25,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c69ec8fb8201481ea20f274de2837d6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681354459172 2023-04-13 02:54:25,451 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3a351c0487b5404088ef7f1331b92bd4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681354459172 2023-04-13 02:54:25,451 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a514d5bf5d37470cb600871a1d7450c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681354461273 2023-04-13 02:54:25,451 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c82b591d25454f5c90ab729e5fd0747d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681354461273 2023-04-13 02:54:25,451 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 48983f893cc24acf946d74caf553d0fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681354461959 2023-04-13 02:54:25,452 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ef3d021b3d284fcf8c74e2bf7290d339, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681354461959 2023-04-13 02:54:25,452 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 87e5348806584453b7fa767ed209edd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463046 2023-04-13 02:54:25,452 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 16bf91dd9fe34574b1d462035f01bb3e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463036 2023-04-13 02:54:25,468 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#B#compaction#162 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:25,503 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#A#compaction#163 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:25,509 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/ca09063086dc4dbe9cc0d2ca68e53e2f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ca09063086dc4dbe9cc0d2ca68e53e2f 2023-04-13 02:54:25,518 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/B of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into ca09063086dc4dbe9cc0d2ca68e53e2f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:25,518 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:25,518 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, priority=9, startTime=1681354465446; duration=0sec 2023-04-13 02:54:25,518 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:25,518 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:54:25,520 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84894 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:54:25,521 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/C is initiating minor compaction (all files) 2023-04-13 02:54:25,521 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/C in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,521 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/5abffb766b5a491a9b692f9d728ee626, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=82.9 K 2023-04-13 02:54:25,522 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5abffb766b5a491a9b692f9d728ee626, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681354455837 2023-04-13 02:54:25,522 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting aeb8c58fb13f47a7813af9bcd13a0341, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1681354455923 2023-04-13 02:54:25,522 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 702f49a5c0a348fcaf5c05fea1dabd50, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681354458006 2023-04-13 02:54:25,523 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b9f8ae104e8744d4ad9817bdd8289c37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681354459172 2023-04-13 02:54:25,523 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 44ad14c923034dd4a1d05b01ca866c11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681354461273 2023-04-13 02:54:25,524 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting acfbfd65fd5048e1b7e711758290f46a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681354461959 2023-04-13 02:54:25,525 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting fc0fbabbcb23411fab3fe9e3c4e9aee1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463046 2023-04-13 02:54:25,549 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:25,550 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#C#compaction#164 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:25,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:25,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:25,558 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:25,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:25,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:25,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:25,561 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:25,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:25,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:25,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:25,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:25,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:25,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:25,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:25,565 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/bb4b4f6747ee48fca60aef69db3a975c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/bb4b4f6747ee48fca60aef69db3a975c 2023-04-13 02:54:25,565 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:25,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:25,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:25,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:25,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:25,569 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:25,569 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:25,571 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/C of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into bb4b4f6747ee48fca60aef69db3a975c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:25,571 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:25,571 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, priority=9, startTime=1681354465446; duration=0sec 2023-04-13 02:54:25,571 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:26,000 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304139dbf192811fd43a6ad340fdc1413b2a6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139dbf192811fd43a6ad340fdc1413b2a6 2023-04-13 02:54:26,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:26,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:26,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354526132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:26,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354526235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:26,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:26,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39710 deadline: 1681354526356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:26,367 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53 2023-04-13 02:54:26,372 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/A of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 8eb7b6c5f32d4b1688d8fcb1ef73bf53(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:26,372 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:26,372 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, priority=9, startTime=1681354465445; duration=0sec 2023-04-13 02:54:26,372 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:26,405 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/8fa7e8e7dc464568bc788723a66bc96a 2023-04-13 02:54:26,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:26,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354526438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:26,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:26,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354526741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:26,819 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/021f2c4e88c34781956aac5de7873a05 2023-04-13 02:54:27,231 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/c8b6f3183bf74ac69ff4c1c2276f8070 2023-04-13 02:54:27,236 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/8fa7e8e7dc464568bc788723a66bc96a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a 2023-04-13 02:54:27,240 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a, entries=150, sequenceid=300, filesize=25.4 K 2023-04-13 02:54:27,241 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/021f2c4e88c34781956aac5de7873a05 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05 2023-04-13 02:54:27,246 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05, entries=150, sequenceid=300, filesize=11.9 K 2023-04-13 02:54:27,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681354527245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,247 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/c8b6f3183bf74ac69ff4c1c2276f8070 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070 2023-04-13 02:54:27,252 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070, entries=150, sequenceid=300, filesize=11.9 K 2023-04-13 02:54:27,252 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1684ms, sequenceid=300, compaction requested=false 2023-04-13 02:54:27,253 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:27,253 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,254 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,255 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6ae210[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,258 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,258 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,258 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,262 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,262 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,262 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,262 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,262 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,271 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,271 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,271 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,271 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,271 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,272 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,272 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:27,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:27,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:27,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:27,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:27,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:27,357 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304138dc9fd34f263450ba159c9d64f570580 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138dc9fd34f263450ba159c9d64f570580 2023-04-13 02:54:27,373 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:27,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39642 deadline: 1681354527371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:27,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39690 deadline: 1681354527372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:27,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:27,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:27,377 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:27,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:27,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:27,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,379 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,379 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,379 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,379 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:27,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:27,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:27,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:27,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:27,382 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,382 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:27,382 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:27,382 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,384 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fb3b9c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,388 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,388 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,391 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,394 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,394 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,394 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,394 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,394 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,396 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,396 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,396 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,397 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,404 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,404 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39642 deadline: 1681354527474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39690 deadline: 1681354527476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,504 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:27,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:27,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:27,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:27,508 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:27,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:27,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:27,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:27,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:27,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:27,512 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,512 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:27,512 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:27,512 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,514 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,514 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,514 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,514 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad9ad7e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,516 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,518 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,518 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,522 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,522 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,523 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,523 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,523 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,529 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,529 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,529 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,629 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:27,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:27,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:27,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:27,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:27,633 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:27,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:27,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:27,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,636 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:27,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:27,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:27,637 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:27,637 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,637 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:27,637 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:27,637 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:27,638 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,638 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:27,638 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:27,638 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,639 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139169e0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,642 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,642 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,642 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,643 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,646 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,646 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,646 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,646 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,646 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,653 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,653 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,654 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,658 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,658 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,658 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,658 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,658 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39642 deadline: 1681354527676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39690 deadline: 1681354527678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,758 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:27,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:27,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:27,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:27,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:27,762 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,762 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,763 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/34475b30a4af46ac910a43c7ab435d3c 2023-04-13 02:54:27,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:27,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:27,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:27,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:27,767 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,767 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:27,767 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:27,767 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,767 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,769 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7673e0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,773 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,775 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a23ecf5319bf478fabfaaf6a252d158a 2023-04-13 02:54:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,775 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,779 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,779 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,783 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,799 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,799 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,899 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:27,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:27,904 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:27,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:27,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:27,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,910 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:27,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:27,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:27,912 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,912 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:27,912 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:27,912 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a89e69[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:27,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:27,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:27,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:27,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:27,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:27,922 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:27,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:27,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:27,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:27,922 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:27,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:27,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:27,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:27,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:27,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:27,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:27,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:27,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:27,929 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:27,930 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:27,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39642 deadline: 1681354527981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:27,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:27,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39690 deadline: 1681354527981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,030 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:28,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:28,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:28,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:28,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:28,033 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:28,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:28,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:28,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:28,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:28,035 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:28,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:28,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:28,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:28,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:28,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:28,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:28,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. started... 2023-04-13 02:54:28,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. as already flushing 2023-04-13 02:54:28,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:28,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:28,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:28,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107d7fbf[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:28,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:28,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:28,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:28,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:28,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:28,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:28,047 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:28,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:28,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:28,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:28,048 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:28,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:28,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:28,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:28,055 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:28,055 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:28,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:28,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:28,056 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:28,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:28,056 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:28,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:28,140 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e0b0a9f to 127.0.0.1:62948 2023-04-13 02:54:28,140 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,141 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1345b761 to 127.0.0.1:62948 2023-04-13 02:54:28,142 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,148 DEBUG [Thread-874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078e8770 to 127.0.0.1:62948 2023-04-13 02:54:28,148 DEBUG [Thread-874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,149 DEBUG [Thread-876] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a77cdd0 to 127.0.0.1:62948 2023-04-13 02:54:28,149 DEBUG [Thread-876] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,156 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:28,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/92e3ec0cc0fe49ee877b4c1134fd1a46 2023-04-13 02:54:28,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/34475b30a4af46ac910a43c7ab435d3c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c 2023-04-13 02:54:28,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c, entries=200, sequenceid=315, filesize=32.3 K 2023-04-13 02:54:28,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/a23ecf5319bf478fabfaaf6a252d158a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a 2023-04-13 02:54:28,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a, entries=150, sequenceid=315, filesize=11.9 K 2023-04-13 02:54:28,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/92e3ec0cc0fe49ee877b4c1134fd1a46 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46 2023-04-13 02:54:28,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46, entries=150, sequenceid=315, filesize=11.9 K 2023-04-13 02:54:28,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 870ms, sequenceid=315, compaction requested=true 2023-04-13 02:54:28,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:28,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:28,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:28,211 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:28,211 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:28,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:28,212 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:28,213 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:28,213 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/A is initiating minor compaction (all files) 2023-04-13 02:54:28,213 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/B is initiating minor compaction (all files) 2023-04-13 02:54:28,213 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/A in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:28,213 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/B in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:28,213 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=83.7 K 2023-04-13 02:54:28,213 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ca09063086dc4dbe9cc0d2ca68e53e2f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=36.3 K 2023-04-13 02:54:28,213 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8eb7b6c5f32d4b1688d8fcb1ef73bf53, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463046 2023-04-13 02:54:28,213 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ca09063086dc4dbe9cc0d2ca68e53e2f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463046 2023-04-13 02:54:28,214 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8fa7e8e7dc464568bc788723a66bc96a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681354464226 2023-04-13 02:54:28,214 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 021f2c4e88c34781956aac5de7873a05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681354464226 2023-04-13 02:54:28,214 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 34475b30a4af46ac910a43c7ab435d3c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681354466115 2023-04-13 02:54:28,214 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a23ecf5319bf478fabfaaf6a252d158a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681354466122 2023-04-13 02:54:28,222 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#B#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:28,225 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#A#compaction#172 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:28,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:28,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-13 02:54:28,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:28,253 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40c57efc to 127.0.0.1:62948 2023-04-13 02:54:28,254 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:28,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:28,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:28,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:28,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:28,361 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bccfdc5 to 127.0.0.1:62948 2023-04-13 02:54:28,361 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,487 DEBUG [Thread-859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x695af552 to 127.0.0.1:62948 2023-04-13 02:54:28,487 DEBUG [Thread-859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,487 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4183389c to 127.0.0.1:62948 2023-04-13 02:54:28,488 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:28,632 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/453d3f66079a4ed8bb4ffa095127d7e4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/453d3f66079a4ed8bb4ffa095127d7e4 2023-04-13 02:54:28,637 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/B of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 453d3f66079a4ed8bb4ffa095127d7e4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:28,637 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:28,637 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, priority=13, startTime=1681354468211; duration=0sec 2023-04-13 02:54:28,637 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:28,637 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:28,638 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:28,638 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): fbff9ea8a72ff1f3ab1b323ef1e5cd25/C is initiating minor compaction (all files) 2023-04-13 02:54:28,638 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fbff9ea8a72ff1f3ab1b323ef1e5cd25/C in TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:28,638 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/bb4b4f6747ee48fca60aef69db3a975c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp, totalSize=36.3 K 2023-04-13 02:54:28,638 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting bb4b4f6747ee48fca60aef69db3a975c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681354463046 2023-04-13 02:54:28,639 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c8b6f3183bf74ac69ff4c1c2276f8070, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681354464226 2023-04-13 02:54:28,639 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 92e3ec0cc0fe49ee877b4c1134fd1a46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681354466122 2023-04-13 02:54:28,651 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): fbff9ea8a72ff1f3ab1b323ef1e5cd25#C#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:28,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133ba40eaed8e44944a3047674a504be3a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133ba40eaed8e44944a3047674a504be3a 2023-04-13 02:54:29,047 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/4aa2bf3c01c347a18745b31bcc1e09f6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/4aa2bf3c01c347a18745b31bcc1e09f6 2023-04-13 02:54:29,051 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/A of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 4aa2bf3c01c347a18745b31bcc1e09f6(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:29,052 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:29,052 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, priority=13, startTime=1681354468211; duration=0sec 2023-04-13 02:54:29,052 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:29,062 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/0189d0c05c1c4b2b9e1574f113bf8da5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0189d0c05c1c4b2b9e1574f113bf8da5 2023-04-13 02:54:29,066 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fbff9ea8a72ff1f3ab1b323ef1e5cd25/C of fbff9ea8a72ff1f3ab1b323ef1e5cd25 into 0189d0c05c1c4b2b9e1574f113bf8da5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:29,066 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:29,066 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25., storeName=fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, priority=13, startTime=1681354468211; duration=0sec 2023-04-13 02:54:29,066 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:29,072 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/6af9218fa2824869a331e8f85b3de592 2023-04-13 02:54:29,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0 2023-04-13 02:54:29,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/44d71971685b4031aa525dfbd59f710e 2023-04-13 02:54:29,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/6af9218fa2824869a331e8f85b3de592 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/6af9218fa2824869a331e8f85b3de592 2023-04-13 02:54:29,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/6af9218fa2824869a331e8f85b3de592, entries=150, sequenceid=336, filesize=25.4 K 2023-04-13 02:54:29,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0 2023-04-13 02:54:29,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0, entries=150, sequenceid=336, filesize=11.9 K 2023-04-13 02:54:29,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/44d71971685b4031aa525dfbd59f710e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44d71971685b4031aa525dfbd59f710e 2023-04-13 02:54:29,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44d71971685b4031aa525dfbd59f710e, entries=150, sequenceid=336, filesize=11.9 K 2023-04-13 02:54:29,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1656ms, sequenceid=336, compaction requested=false 2023-04-13 02:54:29,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:34,186 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x550c7a27 to 127.0.0.1:62948 2023-04-13 02:54:34,186 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:34,186 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5121 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4996 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 2134 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 6402 rows 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 2150 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 6450 rows 2023-04-13 02:54:34,187 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:54:34,187 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39cddc3f to 127.0.0.1:62948 2023-04-13 02:54:34,187 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:34,189 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:54:34,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:54:34,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:34,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:34,194 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354474194"}]},"ts":"1681354474194"} 2023-04-13 02:54:34,195 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:54:34,197 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:54:34,197 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, UNASSIGN}] 2023-04-13 02:54:34,198 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, UNASSIGN 2023-04-13 02:54:34,199 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:34,199 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354474199"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354474199"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354474199"}]},"ts":"1681354474199"} 2023-04-13 02:54:34,200 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:34,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:34,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fbff9ea8a72ff1f3ab1b323ef1e5cd25, disabling compactions & flushes 2023-04-13 02:54:34,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. after waiting 0 ms 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:34,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fbff9ea8a72ff1f3ab1b323ef1e5cd25 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=A 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=B 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fbff9ea8a72ff1f3ab1b323ef1e5cd25, store=C 2023-04-13 02:54:34,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:34,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:34,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134079f798d2e54e7590714aa6b7343370 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134079f798d2e54e7590714aa6b7343370 2023-04-13 02:54:34,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:35,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/f9c3dff05e384e95ba868b1c89b9598a 2023-04-13 02:54:35,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:35,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/0a06b27ce4fb42c3934dcbdeb4cc716c 2023-04-13 02:54:35,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/b697ca3a04f64327851c0c98c993f7cd 2023-04-13 02:54:35,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/A/f9c3dff05e384e95ba868b1c89b9598a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f9c3dff05e384e95ba868b1c89b9598a 2023-04-13 02:54:35,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f9c3dff05e384e95ba868b1c89b9598a, entries=150, sequenceid=346, filesize=25.4 K 2023-04-13 02:54:35,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/B/0a06b27ce4fb42c3934dcbdeb4cc716c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/0a06b27ce4fb42c3934dcbdeb4cc716c 2023-04-13 02:54:36,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/0a06b27ce4fb42c3934dcbdeb4cc716c, entries=150, sequenceid=346, filesize=11.9 K 2023-04-13 02:54:36,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/.tmp/C/b697ca3a04f64327851c0c98c993f7cd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b697ca3a04f64327851c0c98c993f7cd 2023-04-13 02:54:36,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b697ca3a04f64327851c0c98c993f7cd, entries=150, sequenceid=346, filesize=11.9 K 2023-04-13 02:54:36,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for fbff9ea8a72ff1f3ab1b323ef1e5cd25 in 1656ms, sequenceid=346, compaction requested=true 2023-04-13 02:54:36,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f791937b4ea94e50812e2bbf79e38df3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/574fb196753a4a65aaab537a331da76e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/45b5642850a8409baee34609c1b041b1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c] to archive 2023-04-13 02:54:36,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:36,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/bc68a20a2f5b4ea68c955d37b3945f5e 2023-04-13 02:54:36,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3d31041a971d4683aa0941bf14401688 2023-04-13 02:54:36,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f791937b4ea94e50812e2bbf79e38df3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f791937b4ea94e50812e2bbf79e38df3 2023-04-13 02:54:36,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/fba4a3a7b8744e1ebba800a90adc6b35 2023-04-13 02:54:36,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/e18a8be28cc14286b0e62a04c405557a 2023-04-13 02:54:36,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/574fb196753a4a65aaab537a331da76e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/574fb196753a4a65aaab537a331da76e 2023-04-13 02:54:36,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/dd6289030a014d2f899d7faa486ec83e 2023-04-13 02:54:36,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ae834612e42b4a95a64beacd0999f178 2023-04-13 02:54:36,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/21853100f8564794b71c219081fe52e7 2023-04-13 02:54:36,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/d475def9d9b84b21bca040511300a93f 2023-04-13 02:54:36,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/45b5642850a8409baee34609c1b041b1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/45b5642850a8409baee34609c1b041b1 2023-04-13 02:54:36,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8dde9c852c0e45babde5718199f6d84e 2023-04-13 02:54:36,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/81192d2751ae4f3f8fa5bb3581961d15 2023-04-13 02:54:36,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/3a351c0487b5404088ef7f1331b92bd4 2023-04-13 02:54:36,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/c82b591d25454f5c90ab729e5fd0747d 2023-04-13 02:54:36,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/ef3d021b3d284fcf8c74e2bf7290d339 2023-04-13 02:54:36,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/16bf91dd9fe34574b1d462035f01bb3e 2023-04-13 02:54:36,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8eb7b6c5f32d4b1688d8fcb1ef73bf53 2023-04-13 02:54:36,051 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/8fa7e8e7dc464568bc788723a66bc96a 2023-04-13 02:54:36,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/34475b30a4af46ac910a43c7ab435d3c 2023-04-13 02:54:36,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/21f14a81261340f4a0c4d3cc946c88a1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/d74b4b75540d4b7ebba0306fe26e0a4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/9efaf7b74da446c0a61f4842ce6b915a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ca09063086dc4dbe9cc0d2ca68e53e2f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a] to archive 2023-04-13 02:54:36,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:36,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/001042ea6e3b4bc8891e971355d1e9a8 2023-04-13 02:54:36,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/b978343aefef44eead1771e122522823 2023-04-13 02:54:36,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/21f14a81261340f4a0c4d3cc946c88a1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/21f14a81261340f4a0c4d3cc946c88a1 2023-04-13 02:54:36,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ee3229162db646f1b59a0f418e2f3467 2023-04-13 02:54:36,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/40cfbee042454df3a5979dcea2d1e49a 2023-04-13 02:54:36,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/d74b4b75540d4b7ebba0306fe26e0a4f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/d74b4b75540d4b7ebba0306fe26e0a4f 2023-04-13 02:54:36,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23de86bb0de4830bc300b4e7e0f2e3f 2023-04-13 02:54:36,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/db8d35fd22a74f6c8835aa8e075b8149 2023-04-13 02:54:36,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/1d6b98bf580f41678d0d71fcdd31dec2 2023-04-13 02:54:36,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/9efaf7b74da446c0a61f4842ce6b915a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/9efaf7b74da446c0a61f4842ce6b915a 2023-04-13 02:54:36,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/65176bff082c4c5086e84921c5fcc3af 2023-04-13 02:54:36,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/cfa1ae8724b54421b67283466132f03b 2023-04-13 02:54:36,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/6df4e5e394fb44f28514a47228351fba 2023-04-13 02:54:36,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/c69ec8fb8201481ea20f274de2837d6d 2023-04-13 02:54:36,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a514d5bf5d37470cb600871a1d7450c2 2023-04-13 02:54:36,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/48983f893cc24acf946d74caf553d0fe 2023-04-13 02:54:36,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ca09063086dc4dbe9cc0d2ca68e53e2f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/ca09063086dc4dbe9cc0d2ca68e53e2f 2023-04-13 02:54:36,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/87e5348806584453b7fa767ed209edd6 2023-04-13 02:54:36,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/021f2c4e88c34781956aac5de7873a05 2023-04-13 02:54:36,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/a23ecf5319bf478fabfaaf6a252d158a 2023-04-13 02:54:36,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3dd6a9de409e4e7ea47dbb7dae707d03, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/2f821ee4268c476f9257782e503a5e56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/5abffb766b5a491a9b692f9d728ee626, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/bb4b4f6747ee48fca60aef69db3a975c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46] to archive 2023-04-13 02:54:36,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:54:36,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3b3c67c5d1f34c689787759a263fb0e1 2023-04-13 02:54:36,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/73873a9447834a8f8c1367ed4d90dd04 2023-04-13 02:54:36,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3dd6a9de409e4e7ea47dbb7dae707d03 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/3dd6a9de409e4e7ea47dbb7dae707d03 2023-04-13 02:54:36,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/80fe03638d984675855f3840af9a4948 2023-04-13 02:54:36,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/106e726f77cc4e4f893ee5dbadfbc2e5 2023-04-13 02:54:36,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/2f821ee4268c476f9257782e503a5e56 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/2f821ee4268c476f9257782e503a5e56 2023-04-13 02:54:36,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/1ac7efe4411c4f26aa9d0cbdf3e2dd8e 2023-04-13 02:54:36,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/61448e0b13c749d9922a05ba98ad33d8 2023-04-13 02:54:36,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0409d2704ef54c5bb8458bd9e244d69b 2023-04-13 02:54:36,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/5abffb766b5a491a9b692f9d728ee626 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/5abffb766b5a491a9b692f9d728ee626 2023-04-13 02:54:36,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/d55282830858419ca503c58c17e7f8c2 2023-04-13 02:54:36,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/aeb8c58fb13f47a7813af9bcd13a0341 2023-04-13 02:54:36,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/702f49a5c0a348fcaf5c05fea1dabd50 2023-04-13 02:54:36,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b9f8ae104e8744d4ad9817bdd8289c37 2023-04-13 02:54:36,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44ad14c923034dd4a1d05b01ca866c11 2023-04-13 02:54:36,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/acfbfd65fd5048e1b7e711758290f46a 2023-04-13 02:54:36,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/bb4b4f6747ee48fca60aef69db3a975c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/bb4b4f6747ee48fca60aef69db3a975c 2023-04-13 02:54:36,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/fc0fbabbcb23411fab3fe9e3c4e9aee1 2023-04-13 02:54:36,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/c8b6f3183bf74ac69ff4c1c2276f8070 2023-04-13 02:54:36,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/92e3ec0cc0fe49ee877b4c1134fd1a46 2023-04-13 02:54:36,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits/349.seqid, newMaxSeqId=349, maxSeqId=4 2023-04-13 02:54:36,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25. 2023-04-13 02:54:36,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fbff9ea8a72ff1f3ab1b323ef1e5cd25: 2023-04-13 02:54:36,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:36,107 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fbff9ea8a72ff1f3ab1b323ef1e5cd25, regionState=CLOSED 2023-04-13 02:54:36,107 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354476107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354476107"}]},"ts":"1681354476107"} 2023-04-13 02:54:36,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-13 02:54:36,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure fbff9ea8a72ff1f3ab1b323ef1e5cd25, server=jenkins-hbase4.apache.org,34919,1681354416534 in 1.9080 sec 2023-04-13 02:54:36,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-13 02:54:36,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fbff9ea8a72ff1f3ab1b323ef1e5cd25, UNASSIGN in 1.9120 sec 2023-04-13 02:54:36,111 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354476111"}]},"ts":"1681354476111"} 2023-04-13 02:54:36,112 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:54:36,113 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:54:36,115 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9240 sec 2023-04-13 02:54:36,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 02:54:36,298 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-13 02:54:36,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:54:36,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 02:54:36,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,306 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:36,308 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits] 2023-04-13 02:54:36,311 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/4aa2bf3c01c347a18745b31bcc1e09f6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/4aa2bf3c01c347a18745b31bcc1e09f6 2023-04-13 02:54:36,312 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/6af9218fa2824869a331e8f85b3de592 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/6af9218fa2824869a331e8f85b3de592 2023-04-13 02:54:36,313 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f9c3dff05e384e95ba868b1c89b9598a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/A/f9c3dff05e384e95ba868b1c89b9598a 2023-04-13 02:54:36,316 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/0a06b27ce4fb42c3934dcbdeb4cc716c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/0a06b27ce4fb42c3934dcbdeb4cc716c 2023-04-13 02:54:36,317 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/3e262ee1a8c44f33ae4a01ca6dd8f4b0 2023-04-13 02:54:36,318 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/453d3f66079a4ed8bb4ffa095127d7e4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/B/453d3f66079a4ed8bb4ffa095127d7e4 2023-04-13 02:54:36,321 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0189d0c05c1c4b2b9e1574f113bf8da5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/0189d0c05c1c4b2b9e1574f113bf8da5 2023-04-13 02:54:36,322 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44d71971685b4031aa525dfbd59f710e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/44d71971685b4031aa525dfbd59f710e 2023-04-13 02:54:36,323 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b697ca3a04f64327851c0c98c993f7cd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/C/b697ca3a04f64327851c0c98c993f7cd 2023-04-13 02:54:36,326 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits/349.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25/recovered.edits/349.seqid 2023-04-13 02:54:36,326 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/fbff9ea8a72ff1f3ab1b323ef1e5cd25 2023-04-13 02:54:36,326 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:54:36,327 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:54:36,328 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 02:54:36,331 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041304deaf653eb14681841b1c7809591cfa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041304deaf653eb14681841b1c7809591cfa 2023-04-13 02:54:36,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130f497eee8b564d65b59dd57bf0ec5803 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130f497eee8b564d65b59dd57bf0ec5803 2023-04-13 02:54:36,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319242816840a444597f00c570238930d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319242816840a444597f00c570238930d 2023-04-13 02:54:36,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041331d3430c53954cfda60ee272326ee7ab to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041331d3430c53954cfda60ee272326ee7ab 2023-04-13 02:54:36,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133ba40eaed8e44944a3047674a504be3a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133ba40eaed8e44944a3047674a504be3a 2023-04-13 02:54:36,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134079f798d2e54e7590714aa6b7343370 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134079f798d2e54e7590714aa6b7343370 2023-04-13 02:54:36,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344a1c848bcb746e2aec1e64d894e1486 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344a1c848bcb746e2aec1e64d894e1486 2023-04-13 02:54:36,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344c108d586c84f6f98b2f145fc8849ec to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344c108d586c84f6f98b2f145fc8849ec 2023-04-13 02:54:36,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413674797a35dbd4285972d7a4570530a1e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413674797a35dbd4285972d7a4570530a1e 2023-04-13 02:54:36,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041373b354622fac4421b494bb246246e7b3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041373b354622fac4421b494bb246246e7b3 2023-04-13 02:54:36,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137da71b9ad9e44d6ca9df347dac6e10ea to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137da71b9ad9e44d6ca9df347dac6e10ea 2023-04-13 02:54:36,343 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137f7f6b893ddf4eb8a6c2d2098a8684b5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137f7f6b893ddf4eb8a6c2d2098a8684b5 2023-04-13 02:54:36,344 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138dc9fd34f263450ba159c9d64f570580 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138dc9fd34f263450ba159c9d64f570580 2023-04-13 02:54:36,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139b43a8b48ee64e6f958602f7e5145bbb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139b43a8b48ee64e6f958602f7e5145bbb 2023-04-13 02:54:36,346 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139dbf192811fd43a6ad340fdc1413b2a6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139dbf192811fd43a6ad340fdc1413b2a6 2023-04-13 02:54:36,347 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ac0bb3bbf1c24a9682caac5514c90e4e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ac0bb3bbf1c24a9682caac5514c90e4e 2023-04-13 02:54:36,348 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ea1575b26d0a4fb6aa2ecfb6071635aa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ea1575b26d0a4fb6aa2ecfb6071635aa 2023-04-13 02:54:36,349 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fe05ba071b484adda1396dd98b57188d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fe05ba071b484adda1396dd98b57188d 2023-04-13 02:54:36,349 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:54:36,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,353 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:54:36,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:54:36,355 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,356 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:54:36,356 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354476356"}]},"ts":"9223372036854775807"} 2023-04-13 02:54:36,357 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:54:36,357 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fbff9ea8a72ff1f3ab1b323ef1e5cd25, NAME => 'TestAcidGuarantees,,1681354444960.fbff9ea8a72ff1f3ab1b323ef1e5cd25.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:54:36,357 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:54:36,357 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354476357"}]},"ts":"9223372036854775807"} 2023-04-13 02:54:36,359 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:54:36,360 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-04-13 02:54:36,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 02:54:36,402 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-13 02:54:36,415 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231 (was 235), OpenFileDescriptor=534 (was 579), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 463), ProcessCount=169 (was 169), AvailableMemoryMB=886 (was 1060) 2023-04-13 02:54:36,426 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=231, OpenFileDescriptor=534, MaxFileDescriptor=60000, SystemLoadAverage=375, ProcessCount=169, AvailableMemoryMB=886 2023-04-13 02:54:36,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:54:36,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:54:36,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:36,431 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:54:36,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-13 02:54:36,431 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:54:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 02:54:36,432 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:36,433 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c empty. 2023-04-13 02:54:36,433 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:36,433 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:54:36,441 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:54:36,443 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => aa0d51e3ad17c60b2f3a2f2437425a6c, NAME => 'TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:54:36,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 02:54:36,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 02:54:36,848 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:54:36,848 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing aa0d51e3ad17c60b2f3a2f2437425a6c, disabling compactions & flushes 2023-04-13 02:54:36,849 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:36,849 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:36,849 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. after waiting 0 ms 2023-04-13 02:54:36,849 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:36,849 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:36,849 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:36,852 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:54:36,853 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354476853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354476853"}]},"ts":"1681354476853"} 2023-04-13 02:54:36,854 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:54:36,855 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:54:36,855 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354476855"}]},"ts":"1681354476855"} 2023-04-13 02:54:36,856 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:54:36,859 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, ASSIGN}] 2023-04-13 02:54:36,860 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, ASSIGN 2023-04-13 02:54:36,861 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:54:37,012 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=aa0d51e3ad17c60b2f3a2f2437425a6c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:37,012 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354477012"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354477012"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354477012"}]},"ts":"1681354477012"} 2023-04-13 02:54:37,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:37,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 02:54:37,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:37,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aa0d51e3ad17c60b2f3a2f2437425a6c, NAME => 'TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:54:37,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:54:37,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,170 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,172 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A 2023-04-13 02:54:37,172 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A 2023-04-13 02:54:37,172 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:37,172 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa0d51e3ad17c60b2f3a2f2437425a6c columnFamilyName A 2023-04-13 02:54:37,173 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(310): Store=aa0d51e3ad17c60b2f3a2f2437425a6c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:37,173 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,174 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B 2023-04-13 02:54:37,174 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B 2023-04-13 02:54:37,174 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:37,174 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa0d51e3ad17c60b2f3a2f2437425a6c columnFamilyName B 2023-04-13 02:54:37,175 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(310): Store=aa0d51e3ad17c60b2f3a2f2437425a6c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:37,175 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,176 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C 2023-04-13 02:54:37,176 DEBUG [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C 2023-04-13 02:54:37,176 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:54:37,176 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa0d51e3ad17c60b2f3a2f2437425a6c columnFamilyName C 2023-04-13 02:54:37,177 INFO [StoreOpener-aa0d51e3ad17c60b2f3a2f2437425a6c-1] regionserver.HStore(310): Store=aa0d51e3ad17c60b2f3a2f2437425a6c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:54:37,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:37,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:54:37,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:54:37,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened aa0d51e3ad17c60b2f3a2f2437425a6c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64741299, jitterRate=-0.035279467701911926}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:54:37,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:37,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., pid=30, masterSystemTime=1681354477165 2023-04-13 02:54:37,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:37,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:37,185 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=aa0d51e3ad17c60b2f3a2f2437425a6c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:37,185 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354477185"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354477185"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354477185"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354477185"}]},"ts":"1681354477185"} 2023-04-13 02:54:37,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-13 02:54:37,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 in 172 msec 2023-04-13 02:54:37,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-13 02:54:37,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, ASSIGN in 328 msec 2023-04-13 02:54:37,189 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:54:37,189 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354477189"}]},"ts":"1681354477189"} 2023-04-13 02:54:37,190 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:54:37,192 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:54:37,193 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 764 msec 2023-04-13 02:54:37,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 02:54:37,535 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-13 02:54:37,536 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x5b4a16d4 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,541 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13544701, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,542 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,543 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,544 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:54:37,546 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:54:37,547 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1b40c526 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,551 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10b79c2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,551 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x47ae15dc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,555 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a5a81fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,555 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x025522dc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,559 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3562c7bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,559 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x4c6d5986 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,562 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6705d115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,563 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x31182447 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,566 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@179ffa8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,566 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x270a3c0c to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,570 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f29c1c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,570 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1e067bfc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,574 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@681ace6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,575 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x06bbe763 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,578 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b4fa5d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,578 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1c8736d4 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,582 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45de5ad8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,583 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x451c882f to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:54:37,590 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@608837ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:54:37,594 DEBUG [hconnection-0x3f4a8b1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,595 DEBUG [hconnection-0x60a7141a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,595 DEBUG [hconnection-0x306f60da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,597 DEBUG [hconnection-0x63a25938-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,597 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,597 DEBUG [hconnection-0x49037315-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,597 DEBUG [hconnection-0x6262d1bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,597 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,597 DEBUG [hconnection-0x2d7c0317-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,597 DEBUG [hconnection-0x82a470c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,601 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,601 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,603 DEBUG [hconnection-0x5c141a75-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,605 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:37,603 DEBUG [hconnection-0xb3516cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:54:37,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,605 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,607 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,614 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:54:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:37,614 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:37,614 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:37,614 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:37,614 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:37,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:37,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:37,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:37,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:37,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:37,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:37,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:37,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:37,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:37,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:37,619 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:37,619 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:37,619 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:37,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:37,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:37,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:37,634 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/4b72540bb7624f70a136596096fd7b1b 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681354477624, max timestamp=1681354477659Num uniques 859; 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681354477624, max timestamp=1681354477659Num uniques 859; 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681354477624, max timestamp=1681354477659Num uniques 859; 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:54:37,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=7642, suffix heap size=117064,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=167 2023-04-13 02:54:37,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354477684, max timestamp=1681354477690Num uniques 1118; 2023-04-13 02:54:37,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-13 02:54:37,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354477684, max timestamp=1681354477690Num uniques 1026; 2023-04-13 02:54:37,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12268, suffix heap size=134672,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=268 2023-04-13 02:54:37,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681354477684, max timestamp=1681354477690Num uniques 1118; 2023-04-13 02:54:37,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-13 02:54:37,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681354477719, max timestamp=1681354477731Num uniques 1127; 2023-04-13 02:54:37,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51610, new segment data size=6960, suffix heap size=145176,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=152 2023-04-13 02:54:37,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681354477719, max timestamp=1681354477730Num uniques 1127; 2023-04-13 02:54:37,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681354477719, max timestamp=1681354477730Num uniques 1127; 2023-04-13 02:54:37,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 02:54:37,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-13 02:54:37,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354477749, max timestamp=1681354477754Num uniques 1011; 2023-04-13 02:54:37,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46306, new segment data size=6870, suffix heap size=133112,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=150 2023-04-13 02:54:37,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354477749, max timestamp=1681354477754Num uniques 1086; 2023-04-13 02:54:37,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 02:54:37,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354477749, max timestamp=1681354477754Num uniques 1086; 2023-04-13 02:54:37,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681354477767, max timestamp=1681354477772Num uniques 1045; 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681354477767, max timestamp=1681354477772Num uniques 1045; 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681354477767, max timestamp=1681354477772Num uniques 1009; 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-13 02:54:37,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13510, suffix heap size=132904,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=295 2023-04-13 02:54:37,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681354477785, max timestamp=1681354477792Num uniques 1154; 2023-04-13 02:54:37,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-13 02:54:37,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681354477785, max timestamp=1681354477792Num uniques 1154; 2023-04-13 02:54:37,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681354477785, max timestamp=1681354477792Num uniques 1154; 2023-04-13 02:54:37,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-13 02:54:37,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681354477799, max timestamp=1681354477810Num uniques 1113; 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681354477799, max timestamp=1681354477810Num uniques 1113; 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681354477799, max timestamp=1681354477810Num uniques 1113; 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-13 02:54:37,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-13 02:54:37,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7866, totalHeapSize=18128, min timestamp=1681354477819, max timestamp=1681354477823Num uniques 1122; 2023-04-13 02:54:37,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51376, new segment data size=7866, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-13 02:54:37,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681354477819, max timestamp=1681354477823Num uniques 1122; 2023-04-13 02:54:37,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681354477819, max timestamp=1681354477823Num uniques 1122; 2023-04-13 02:54:37,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-13 02:54:37,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-13 02:54:37,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1681354477834, max timestamp=1681354477838Num uniques 1031; 2023-04-13 02:54:37,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8279, totalHeapSize=19064, min timestamp=1681354477834, max timestamp=1681354477838Num uniques 1031; 2023-04-13 02:54:37,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47206, new segment data size=8276, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:54:37,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47209, new segment data size=8279, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:54:37,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8278, totalHeapSize=19064, min timestamp=1681354477834, max timestamp=1681354477838Num uniques 1031; 2023-04-13 02:54:37,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47208, new segment data size=8278, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 02:54:37,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681354477845, max timestamp=1681354477851Num uniques 1040; 2023-04-13 02:54:37,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-13 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681354477845, max timestamp=1681354477851Num uniques 1040; 2023-04-13 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-13 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681354477845, max timestamp=1681354477851Num uniques 1040; 2023-04-13 02:54:37,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-04-13 02:54:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681354477863, max timestamp=1681354477866Num uniques 1199; 2023-04-13 02:54:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-13 02:54:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681354477863, max timestamp=1681354477866Num uniques 1199; 2023-04-13 02:54:37,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-13 02:54:37,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681354477863, max timestamp=1681354477866Num uniques 1199; 2023-04-13 02:54:37,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1681354477871, max timestamp=1681354477897Num uniques 1058; 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1681354477871, max timestamp=1681354477897Num uniques 1058; 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1681354477871, max timestamp=1681354477897Num uniques 1058; 2023-04-13 02:54:37,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681354477911, max timestamp=1681354477912Num uniques 1067; 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681354477911, max timestamp=1681354477912Num uniques 1067; 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681354477911, max timestamp=1681354477912Num uniques 1067; 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 02:54:37,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 02:54:37,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681354477925, max timestamp=1681354477928Num uniques 1026; 2023-04-13 02:54:37,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-13 02:54:37,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681354477925, max timestamp=1681354477928Num uniques 1026; 2023-04-13 02:54:37,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-13 02:54:37,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354477925, max timestamp=1681354477928Num uniques 1026; 2023-04-13 02:54:37,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=6870, suffix heap size=134672,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=150 2023-04-13 02:54:37,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681354477940, max timestamp=1681354477944Num uniques 1085; 2023-04-13 02:54:37,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-13 02:54:37,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681354477940, max timestamp=1681354477944Num uniques 1085; 2023-04-13 02:54:37,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-13 02:54:37,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681354477940, max timestamp=1681354477945Num uniques 1009; 2023-04-13 02:54:37,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8470, suffix heap size=132904,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=185 2023-04-13 02:54:37,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681354477956, max timestamp=1681354477960Num uniques 1044; 2023-04-13 02:54:37,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 02:54:37,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681354477956, max timestamp=1681354477960Num uniques 1044; 2023-04-13 02:54:37,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 02:54:37,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681354477956, max timestamp=1681354477960Num uniques 1044; 2023-04-13 02:54:37,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-13 02:54:37,966 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 02:54:37,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681354477973, max timestamp=1681354477979Num uniques 1053; 2023-04-13 02:54:37,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681354477973, max timestamp=1681354477979Num uniques 1053; 2023-04-13 02:54:37,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681354477973, max timestamp=1681354477979Num uniques 1053; 2023-04-13 02:54:37,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-13 02:54:37,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-13 02:54:37,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-13 02:54:37,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681354477985, max timestamp=1681354477995Num uniques 1062; 2023-04-13 02:54:37,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=14282, suffix heap size=138416,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=312 2023-04-13 02:54:37,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681354477985, max timestamp=1681354477995Num uniques 1062; 2023-04-13 02:54:37,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=14282, suffix heap size=138416,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=312 2023-04-13 02:54:37,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681354477985, max timestamp=1681354477995Num uniques 1062; 2023-04-13 02:54:37,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=14282, suffix heap size=138416,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=312 2023-04-13 02:54:38,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681354478002, max timestamp=1681354478007Num uniques 1171; 2023-04-13 02:54:38,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-04-13 02:54:38,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681354478002, max timestamp=1681354478007Num uniques 1171; 2023-04-13 02:54:38,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-04-13 02:54:38,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681354478002, max timestamp=1681354478007Num uniques 1171; 2023-04-13 02:54:38,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=7826, suffix heap size=149752,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=171 2023-04-13 02:54:38,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681354478015, max timestamp=1681354478020Num uniques 1030; 2023-04-13 02:54:38,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-13 02:54:38,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681354478018, max timestamp=1681354478020Num uniques 1030; 2023-04-13 02:54:38,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 02:54:38,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478018, max timestamp=1681354478020Num uniques 1030; 2023-04-13 02:54:38,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=6870, suffix heap size=135088,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=150 2023-04-13 02:54:38,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681354478028, max timestamp=1681354478037Num uniques 1089; 2023-04-13 02:54:38,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-13 02:54:38,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478028, max timestamp=1681354478038Num uniques 1039; 2023-04-13 02:54:38,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47574, new segment data size=6870, suffix heap size=136024,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=150 2023-04-13 02:54:38,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478028, max timestamp=1681354478038Num uniques 1009; 2023-04-13 02:54:38,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 02:54:38,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681354478047, max timestamp=1681354478051Num uniques 1009; 2023-04-13 02:54:38,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=11358, suffix heap size=132904,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=248 2023-04-13 02:54:38,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681354478047, max timestamp=1681354478051Num uniques 1048; 2023-04-13 02:54:38,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 02:54:38,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478050, max timestamp=1681354478051Num uniques 1009; 2023-04-13 02:54:38,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 02:54:38,059 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/fe3a7203629b4bbca2f2ba45dcb3db15 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681354478087, max timestamp=1681354478091Num uniques 1107; 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1681354478087, max timestamp=1681354478091Num uniques 1009; 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=6915, suffix heap size=132904,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=151 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681354478087, max timestamp=1681354478091Num uniques 1107; 2023-04-13 02:54:38,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7590, totalHeapSize=17504, min timestamp=1681354478095, max timestamp=1681354478104Num uniques 1010; 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7590, suffix heap size=133008,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=166 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681354478095, max timestamp=1681354478103Num uniques 1016; 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=7596, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681354478095, max timestamp=1681354478103Num uniques 1016; 2023-04-13 02:54:38,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=7596, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-04-13 02:54:38,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681354478113, max timestamp=1681354478116Num uniques 1025; 2023-04-13 02:54:38,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-13 02:54:38,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681354478113, max timestamp=1681354478116Num uniques 1025; 2023-04-13 02:54:38,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-13 02:54:38,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681354478113, max timestamp=1681354478116Num uniques 1025; 2023-04-13 02:54:38,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-13 02:54:38,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681354478125, max timestamp=1681354478129Num uniques 1084; 2023-04-13 02:54:38,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-13 02:54:38,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681354478125, max timestamp=1681354478129Num uniques 1084; 2023-04-13 02:54:38,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681354478125, max timestamp=1681354478129Num uniques 1084; 2023-04-13 02:54:38,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-13 02:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681354478140, max timestamp=1681354478143Num uniques 1093; 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681354478140, max timestamp=1681354478143Num uniques 1093; 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681354478140, max timestamp=1681354478143Num uniques 1093; 2023-04-13 02:54:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 02:54:38,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681354478154, max timestamp=1681354478159Num uniques 1102; 2023-04-13 02:54:38,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681354478154, max timestamp=1681354478160Num uniques 1102; 2023-04-13 02:54:38,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1681354478154, max timestamp=1681354478160Num uniques 1102; 2023-04-13 02:54:38,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-13 02:54:38,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-13 02:54:38,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-13 02:54:38,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681354478175, max timestamp=1681354478177Num uniques 1011; 2023-04-13 02:54:38,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-04-13 02:54:38,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681354478175, max timestamp=1681354478177Num uniques 1011; 2023-04-13 02:54:38,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681354478175, max timestamp=1681354478177Num uniques 1011; 2023-04-13 02:54:38,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-04-13 02:54:38,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7771, totalHeapSize=17920, min timestamp=1681354478188, max timestamp=1681354478191Num uniques 1020; 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1681354478188, max timestamp=1681354478191Num uniques 1020; 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46701, new segment data size=7771, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7770, totalHeapSize=17920, min timestamp=1681354478188, max timestamp=1681354478191Num uniques 1020; 2023-04-13 02:54:38,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46700, new segment data size=7770, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 02:54:38,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8186, totalHeapSize=18856, min timestamp=1681354478202, max timestamp=1681354478206Num uniques 1029; 2023-04-13 02:54:38,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47116, new segment data size=8186, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-13 02:54:38,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8188, totalHeapSize=18856, min timestamp=1681354478202, max timestamp=1681354478206Num uniques 1029; 2023-04-13 02:54:38,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47118, new segment data size=8188, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-13 02:54:38,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478202, max timestamp=1681354478206Num uniques 1029; 2023-04-13 02:54:38,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47119, new segment data size=6870, suffix heap size=134984,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=150 2023-04-13 02:54:38,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10888, totalHeapSize=24992, min timestamp=1681354478216, max timestamp=1681354478221Num uniques 1038; 2023-04-13 02:54:38,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47528, new segment data size=10888, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-13 02:54:38,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10888, totalHeapSize=24992, min timestamp=1681354478216, max timestamp=1681354478221Num uniques 1038; 2023-04-13 02:54:38,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47528, new segment data size=10888, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-13 02:54:38,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8600, totalHeapSize=19792, min timestamp=1681354478219, max timestamp=1681354478221Num uniques 1009; 2023-04-13 02:54:38,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=8600, suffix heap size=132904,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=188 2023-04-13 02:54:38,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681354478231, max timestamp=1681354478260Num uniques 1097; 2023-04-13 02:54:38,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-13 02:54:38,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681354478231, max timestamp=1681354478260Num uniques 1097; 2023-04-13 02:54:38,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-13 02:54:38,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681354478231, max timestamp=1681354478260Num uniques 1047; 2023-04-13 02:54:38,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=11312, suffix heap size=136856,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=247 2023-04-13 02:54:38,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681354478273, max timestamp=1681354478278Num uniques 1106; 2023-04-13 02:54:38,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-13 02:54:38,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681354478273, max timestamp=1681354478278Num uniques 1106; 2023-04-13 02:54:38,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-13 02:54:38,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681354478273, max timestamp=1681354478278Num uniques 1106; 2023-04-13 02:54:38,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1681354478279, max timestamp=1681354478294Num uniques 1015; 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1681354478279, max timestamp=1681354478294Num uniques 1015; 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1681354478279, max timestamp=1681354478294Num uniques 1015; 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=9840, suffix heap size=133528,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=215 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=9840, suffix heap size=133528,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=215 2023-04-13 02:54:38,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=9840, suffix heap size=133528,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=215 2023-04-13 02:54:38,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681354478302, max timestamp=1681354478306Num uniques 1074; 2023-04-13 02:54:38,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-04-13 02:54:38,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681354478302, max timestamp=1681354478306Num uniques 1074; 2023-04-13 02:54:38,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-04-13 02:54:38,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681354478302, max timestamp=1681354478306Num uniques 1074; 2023-04-13 02:54:38,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-04-13 02:54:38,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354538320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8368, totalHeapSize=19272, min timestamp=1681354478318, max timestamp=1681354478321Num uniques 1083; 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8370, totalHeapSize=19272, min timestamp=1681354478318, max timestamp=1681354478321Num uniques 1083; 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49588, new segment data size=8368, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49590, new segment data size=8370, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8368, totalHeapSize=19272, min timestamp=1681354478318, max timestamp=1681354478321Num uniques 1083; 2023-04-13 02:54:38,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49588, new segment data size=8368, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 02:54:38,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1681354478326, max timestamp=1681354478334Num uniques 1042; 2023-04-13 02:54:38,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-04-13 02:54:38,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1681354478326, max timestamp=1681354478334Num uniques 1042; 2023-04-13 02:54:38,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-04-13 02:54:38,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1681354478326, max timestamp=1681354478334Num uniques 1042; 2023-04-13 02:54:38,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-04-13 02:54:38,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=301, cellSize=13776, totalHeapSize=31544, min timestamp=1681354478342, max timestamp=1681354478351Num uniques 1151; 2023-04-13 02:54:38,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52706, new segment data size=13776, suffix heap size=147672,new segment heap size=31544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=301 2023-04-13 02:54:38,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=301, cellSize=13776, totalHeapSize=31544, min timestamp=1681354478342, max timestamp=1681354478351Num uniques 1151; 2023-04-13 02:54:38,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52706, new segment data size=13776, suffix heap size=147672,new segment heap size=31544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=301 2023-04-13 02:54:38,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=301, cellSize=13776, totalHeapSize=31544, min timestamp=1681354478342, max timestamp=1681354478351Num uniques 1151; 2023-04-13 02:54:38,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52706, new segment data size=13776, suffix heap size=147672,new segment heap size=31544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=301 2023-04-13 02:54:38,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681354478369, max timestamp=1681354478375Num uniques 1160; 2023-04-13 02:54:38,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53120, new segment data size=7320, suffix heap size=148608,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1160, new segment cells count=160 2023-04-13 02:54:38,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681354478369, max timestamp=1681354478375Num uniques 1160; 2023-04-13 02:54:38,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1681354478369, max timestamp=1681354478375Num uniques 1160; 2023-04-13 02:54:38,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53120, new segment data size=7320, suffix heap size=148608,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1160, new segment cells count=160 2023-04-13 02:54:38,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53120, new segment data size=7320, suffix heap size=148608,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1160, new segment cells count=160 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681354478384, max timestamp=1681354478392Num uniques 1019; 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681354478384, max timestamp=1681354478392Num uniques 1019; 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1681354478384, max timestamp=1681354478392Num uniques 1019; 2023-04-13 02:54:38,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-04-13 02:54:38,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681354478398, max timestamp=1681354478407Num uniques 1128; 2023-04-13 02:54:38,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-13 02:54:38,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681354478398, max timestamp=1681354478407Num uniques 1128; 2023-04-13 02:54:38,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-13 02:54:38,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681354478398, max timestamp=1681354478407Num uniques 1128; 2023-04-13 02:54:38,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-13 02:54:38,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681354478419, max timestamp=1681354478423Num uniques 1087; 2023-04-13 02:54:38,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=8562, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-04-13 02:54:38,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681354478419, max timestamp=1681354478423Num uniques 1087; 2023-04-13 02:54:38,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=8562, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-04-13 02:54:38,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8560, totalHeapSize=19688, min timestamp=1681354478419, max timestamp=1681354478424Num uniques 1087; 2023-04-13 02:54:38,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49780, new segment data size=8560, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-04-13 02:54:38,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681354478432, max timestamp=1681354478436Num uniques 1046; 2023-04-13 02:54:38,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-13 02:54:38,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681354478432, max timestamp=1681354478436Num uniques 1046; 2023-04-13 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-13 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681354478432, max timestamp=1681354478436Num uniques 1046; 2023-04-13 02:54:38,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681354478441, max timestamp=1681354478449Num uniques 1105; 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=11675, suffix heap size=142888,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=255 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681354478441, max timestamp=1681354478449Num uniques 1105; 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=255, cellSize=11675, totalHeapSize=26760, min timestamp=1681354478441, max timestamp=1681354478449Num uniques 1105; 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=11675, suffix heap size=142888,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=255 2023-04-13 02:54:38,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=11675, suffix heap size=142888,new segment heap size=26760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=255 2023-04-13 02:54:38,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681354478457, max timestamp=1681354478464Num uniques 1114; 2023-04-13 02:54:38,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 02:54:38,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681354478457, max timestamp=1681354478464Num uniques 1114; 2023-04-13 02:54:38,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 02:54:38,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1681354478457, max timestamp=1681354478464Num uniques 1114; 2023-04-13 02:54:38,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51014, new segment data size=9794, suffix heap size=143824,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1114, new segment cells count=214 2023-04-13 02:54:38,477 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/45480208791b42a2a74acac7f497229b 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=273, cellSize=12498, totalHeapSize=28632, min timestamp=1681354478474, max timestamp=1681354478478Num uniques 1073; 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=12498, suffix heap size=139560,new segment heap size=28632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=273 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=273, cellSize=12498, totalHeapSize=28632, min timestamp=1681354478474, max timestamp=1681354478478Num uniques 1073; 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=12498, suffix heap size=139560,new segment heap size=28632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=273 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=273, cellSize=12498, totalHeapSize=28632, min timestamp=1681354478474, max timestamp=1681354478478Num uniques 1073; 2023-04-13 02:54:38,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=12498, suffix heap size=139560,new segment heap size=28632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=273 2023-04-13 02:54:38,486 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/4b72540bb7624f70a136596096fd7b1b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b 2023-04-13 02:54:38,490 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b, entries=100, sequenceid=6, filesize=9.3 K 2023-04-13 02:54:38,491 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/fe3a7203629b4bbca2f2ba45dcb3db15 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15 2023-04-13 02:54:38,495 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15, entries=100, sequenceid=6, filesize=9.3 K 2023-04-13 02:54:38,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354478490, max timestamp=1681354478494Num uniques 1132; 2023-04-13 02:54:38,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1681354478486, max timestamp=1681354478494Num uniques 1132; 2023-04-13 02:54:38,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51842, new segment data size=6870, suffix heap size=145696,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1132, new segment cells count=150 2023-04-13 02:54:38,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51842, new segment data size=10622, suffix heap size=145696,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1132, new segment cells count=232 2023-04-13 02:54:38,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1681354478486, max timestamp=1681354478494Num uniques 1132; 2023-04-13 02:54:38,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51842, new segment data size=10622, suffix heap size=145696,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1132, new segment cells count=232 2023-04-13 02:54:38,498 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/45480208791b42a2a74acac7f497229b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b 2023-04-13 02:54:38,513 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b, entries=100, sequenceid=6, filesize=9.3 K 2023-04-13 02:54:38,515 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=123.81 KB/126778 for aa0d51e3ad17c60b2f3a2f2437425a6c in 895ms, sequenceid=6, compaction requested=false 2023-04-13 02:54:38,515 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:38,515 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:38,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:38,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=130.52 KB heapSize=369.81 KB 2023-04-13 02:54:38,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1681354478511, max timestamp=1681354478515Num uniques 1009; 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8746, suffix heap size=132904,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=191 2023-04-13 02:54:38,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,518 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=241, cellSize=11036, totalHeapSize=25304, min timestamp=1681354478507, max timestamp=1681354478515Num uniques 1091; 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49966, new segment data size=11036, suffix heap size=141432,new segment heap size=25304 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1091, new segment cells count=241 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1681354478511, max timestamp=1681354478515Num uniques 1091; 2023-04-13 02:54:38,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49966, new segment data size=8746, suffix heap size=141432,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1091, new segment cells count=191 2023-04-13 02:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:38,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83bca39[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:38,520 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:38,520 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:38,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:38,521 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:38,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:38,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:38,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:38,523 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,525 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:38,525 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,525 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:38,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:38,527 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:38,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:38,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:38,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:38,527 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:38,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:38,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:38,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:38,537 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:38,537 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:38,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/f386d51c655c43a6b1d74deb349e8b17 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354478536, max timestamp=1681354478542Num uniques 859; 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354478536, max timestamp=1681354478542Num uniques 859; 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354478536, max timestamp=1681354478542Num uniques 859; 2023-04-13 02:54:38,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:38,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354478560, max timestamp=1681354478567Num uniques 1018; 2023-04-13 02:54:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354538566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-13 02:54:38,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354478560, max timestamp=1681354478567Num uniques 1018; 2023-04-13 02:54:38,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-13 02:54:38,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681354478560, max timestamp=1681354478567Num uniques 1018; 2023-04-13 02:54:38,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-13 02:54:38,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354538582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354538583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354538583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354538584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,637 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:38,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:38,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:38,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:38,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:38,640 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:38,640 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,640 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:38,640 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:38,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,642 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,642 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,643 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,643 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:38,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:38,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:38,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:38,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:38,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:38,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:38,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:38,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:38,646 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ef41ad[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:38,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:38,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:38,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:38,653 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:38,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:38,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:38,655 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:38,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:38,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:38,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:38,656 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:38,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,658 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:38,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,661 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:38,661 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,661 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354538671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354538687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354538687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354538687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354538688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,762 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:38,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:38,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:38,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:38,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:38,765 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:38,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:38,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:38,765 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,767 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,767 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,767 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:38,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:38,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:38,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:38,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:38,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:38,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:38,769 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:38,774 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,775 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,775 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:38,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@687daaf[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:38,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:38,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,777 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:38,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:38,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:38,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:38,779 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:38,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:38,782 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:38,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:38,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:38,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:38,782 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:38,783 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,783 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,783 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,789 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:38,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:38,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,789 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:38,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,789 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354538874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,889 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:38,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:38,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:38,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354538890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354538891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:38,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354538891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,893 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:38,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:38,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:38,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354538891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:38,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:38,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:38,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:38,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:38,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:38,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:38,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:38,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:38,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:38,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:38,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfccd47[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:38,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:38,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:38,905 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:38,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:38,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:38,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:38,909 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:38,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:38,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:38,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:38,909 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:38,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:38,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,914 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:38,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:38,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:38,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:38,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:38,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:38,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:38,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:38,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:38,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,918 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:38,918 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:38,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:38,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:38,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:38,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:38,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/fb1d1d68139147b6a21787fb7ab867b2 2023-04-13 02:54:38,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/02d47cd8fb6840a1a9c1eb905879b03a 2023-04-13 02:54:38,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/f386d51c655c43a6b1d74deb349e8b17 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17 2023-04-13 02:54:38,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17, entries=150, sequenceid=852, filesize=11.9 K 2023-04-13 02:54:38,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/fb1d1d68139147b6a21787fb7ab867b2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2 2023-04-13 02:54:39,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2, entries=150, sequenceid=852, filesize=11.9 K 2023-04-13 02:54:39,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/02d47cd8fb6840a1a9c1eb905879b03a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a 2023-04-13 02:54:39,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a, entries=150, sequenceid=852, filesize=11.9 K 2023-04-13 02:54:39,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~35.78 KB/36640, heapSize ~88.19 KB/90304, currentSize=154.31 KB/158010 for aa0d51e3ad17c60b2f3a2f2437425a6c in 489ms, sequenceid=852, compaction requested=false 2023-04-13 02:54:39,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:39,018 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:39,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:39,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:39,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:39,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:39,021 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:39,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:39,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:39,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:39,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:39,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:39,023 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:39,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:39,024 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:39,024 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:39,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:39,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:39,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:39,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:39,025 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:39,025 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:39,025 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:39,025 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:39,025 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:39,025 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:39,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:39,026 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:39,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:39,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:39,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354539193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354539194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354539195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354539196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354539196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354539298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,444 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/21ca1cf8dc584776a5a7f361b64adb80 2023-04-13 02:54:39,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354539501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354539697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354539698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354539700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354539701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:39,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354539805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:39,855 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/b4ee9ce07aa94ed6a7476c1ceb781aaf 2023-04-13 02:54:40,267 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/1ce31a1465b54dd6ae1a6448280371cd 2023-04-13 02:54:40,272 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/21ca1cf8dc584776a5a7f361b64adb80 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80 2023-04-13 02:54:40,277 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80, entries=150, sequenceid=906, filesize=11.9 K 2023-04-13 02:54:40,277 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-13 02:54:40,278 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/b4ee9ce07aa94ed6a7476c1ceb781aaf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf 2023-04-13 02:54:40,282 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf, entries=150, sequenceid=906, filesize=11.9 K 2023-04-13 02:54:40,282 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/1ce31a1465b54dd6ae1a6448280371cd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd 2023-04-13 02:54:40,286 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd, entries=150, sequenceid=906, filesize=11.9 K 2023-04-13 02:54:40,287 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for aa0d51e3ad17c60b2f3a2f2437425a6c in 1262ms, sequenceid=906, compaction requested=true 2023-04-13 02:54:40,287 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:40,287 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,289 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,289 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc0aac3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,296 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,297 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,303 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:40,303 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,303 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:40,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:40,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:40,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/95bbe55a8ca94663b0e4c599f4e579f9 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354480360, max timestamp=1681354480372Num uniques 859; 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354480360, max timestamp=1681354480372Num uniques 859; 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354480360, max timestamp=1681354480372Num uniques 859; 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:40,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:40,404 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:40,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:40,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:40,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:40,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:40,407 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:40,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:40,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:40,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,408 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,409 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:40,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:40,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:40,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:40,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,413 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ed643f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,419 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,419 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,420 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,423 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,423 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,425 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,429 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,429 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354540431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,530 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:40,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:40,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:40,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:40,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:40,533 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:40,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:40,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:40,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:40,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:40,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:40,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:40,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354540535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,537 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,537 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:40,537 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:40,538 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,539 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0788af[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,552 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,552 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,558 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:40,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,558 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,659 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:40,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:40,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:40,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:40,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:40,662 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:40,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:40,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:40,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:40,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:40,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:40,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:40,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:40,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470ae09d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,691 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,691 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,695 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,699 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,699 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354540704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354540705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354540707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354540707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:40,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354540738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/bd3ad1e3e41a42be8581a489e92ff94c 2023-04-13 02:54:40,799 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:40,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:40,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:40,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:40,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:40,802 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:40,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:40,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:40,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:40,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:40,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:40,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:40,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:40,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b2dc0d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,820 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,822 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:40,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,822 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,823 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,823 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,923 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:40,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:40,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:40,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:40,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:40,926 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:40,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:40,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:40,927 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,927 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:40,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:40,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:40,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:40,931 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,931 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:40,931 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:40,931 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:40,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:40,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f19198[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:40,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:40,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:40,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:40,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:40,943 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:40,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:40,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:40,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:40,943 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:40,945 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,945 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:40,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:40,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:40,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:40,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:40,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:40,951 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:40,952 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:40,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:40,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:40,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:41,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:41,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354541044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,052 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:41,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:41,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:41,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:41,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:41,055 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:41,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:41,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:41,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:41,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:41,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:41,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:41,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:41,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:41,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:41,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:41,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:41,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:41,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d32f19[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:41,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:41,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:41,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:41,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:41,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:41,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:41,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:41,068 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:41,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:41,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:41,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:41,068 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:41,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:41,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:41,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:41,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:41,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:41,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,075 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:41,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:41,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,075 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:41,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:41,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/1c965f00271647b39a2cb8bd5836e9a0 2023-04-13 02:54:41,175 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:41,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/95bbe55a8ca94663b0e4c599f4e579f9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9 2023-04-13 02:54:41,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:41,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:41,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:41,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:41,179 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:41,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:41,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:41,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9, entries=200, sequenceid=916, filesize=14.3 K 2023-04-13 02:54:41,181 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,181 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:41,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:41,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/bd3ad1e3e41a42be8581a489e92ff94c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c 2023-04-13 02:54:41,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:41,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:41,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:41,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:41,183 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,183 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:41,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:41,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:41,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164d2956[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:41,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:41,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c, entries=150, sequenceid=916, filesize=11.9 K 2023-04-13 02:54:41,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/1c965f00271647b39a2cb8bd5836e9a0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0 2023-04-13 02:54:41,191 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:41,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:41,194 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0, entries=150, sequenceid=916, filesize=11.9 K 2023-04-13 02:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for aa0d51e3ad17c60b2f3a2f2437425a6c in 882ms, sequenceid=916, compaction requested=true 2023-04-13 02:54:41,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:41,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:54:41,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 02:54:41,197 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,197 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:41,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:41,199 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:41,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:41,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:41,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:41,201 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:41,201 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:41,201 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:41,201 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,201 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:41,201 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=47.4 K 2023-04-13 02:54:41,201 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:41,201 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,201 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=45.0 K 2023-04-13 02:54:41,202 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4b72540bb7624f70a136596096fd7b1b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681354477614 2023-04-13 02:54:41,202 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 45480208791b42a2a74acac7f497229b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681354477614 2023-04-13 02:54:41,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:41,203 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 02d47cd8fb6840a1a9c1eb905879b03a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1681354478511 2023-04-13 02:54:41,202 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f386d51c655c43a6b1d74deb349e8b17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1681354478511 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:41,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:41,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,204 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 21ca1cf8dc584776a5a7f361b64adb80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1681354478582 2023-04-13 02:54:41,204 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1ce31a1465b54dd6ae1a6448280371cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1681354478582 2023-04-13 02:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,204 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 95bbe55a8ca94663b0e4c599f4e579f9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:41,204 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1c965f00271647b39a2cb8bd5836e9a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,209 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:41,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:41,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:41,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:41,210 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:41,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,210 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:41,210 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:41,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:41,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:41,219 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:41,225 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:41,241 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/e5858e13ca04441c82daa7dc502d6a3c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e5858e13ca04441c82daa7dc502d6a3c 2023-04-13 02:54:41,246 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into e5858e13ca04441c82daa7dc502d6a3c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:41,246 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:41,246 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=12, startTime=1681354481197; duration=0sec 2023-04-13 02:54:41,246 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:41,246 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:41,247 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:41,247 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:41,248 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,248 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=45.0 K 2023-04-13 02:54:41,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting fe3a7203629b4bbca2f2ba45dcb3db15, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681354477614 2023-04-13 02:54:41,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting fb1d1d68139147b6a21787fb7ab867b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1681354478511 2023-04-13 02:54:41,249 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b4ee9ce07aa94ed6a7476c1ceb781aaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=906, earliestPutTs=1681354478582 2023-04-13 02:54:41,249 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting bd3ad1e3e41a42be8581a489e92ff94c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:41,256 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:41,310 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:41,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:41,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:41,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:41,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:41,314 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:41,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:41,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:41,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:41,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:41,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:41,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:41,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:41,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:41,318 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:41,318 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:41,318 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:41,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:41,327 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/d324f7837d4d49f8b177f64aa9d8f596 2023-04-13 02:54:41,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:41,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:41,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:41,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354541573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,635 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:54:41,641 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/c86b53ec13e8474797d749705b30464d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c86b53ec13e8474797d749705b30464d 2023-04-13 02:54:41,650 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into c86b53ec13e8474797d749705b30464d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:41,650 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:41,650 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=12, startTime=1681354481197; duration=0sec 2023-04-13 02:54:41,650 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:41,667 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/be1bb2ad28414011886dd484294a6032 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/be1bb2ad28414011886dd484294a6032 2023-04-13 02:54:41,672 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into be1bb2ad28414011886dd484294a6032(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:41,672 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:41,672 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=12, startTime=1681354481197; duration=0sec 2023-04-13 02:54:41,672 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:41,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354541676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:41,738 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/8073a0fe3edb4e12a1fd75b7d34d516b 2023-04-13 02:54:41,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:41,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354541878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,172 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/c5c144793df44fcabff01ab9f701e883 2023-04-13 02:54:42,177 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/d324f7837d4d49f8b177f64aa9d8f596 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596 2023-04-13 02:54:42,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354542181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,182 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596, entries=150, sequenceid=954, filesize=11.9 K 2023-04-13 02:54:42,183 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/8073a0fe3edb4e12a1fd75b7d34d516b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b 2023-04-13 02:54:42,201 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b, entries=150, sequenceid=954, filesize=11.9 K 2023-04-13 02:54:42,202 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/c5c144793df44fcabff01ab9f701e883 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883 2023-04-13 02:54:42,208 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883, entries=150, sequenceid=954, filesize=11.9 K 2023-04-13 02:54:42,209 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for aa0d51e3ad17c60b2f3a2f2437425a6c in 891ms, sequenceid=954, compaction requested=false 2023-04-13 02:54:42,209 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:42,209 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,209 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,209 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,209 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,214 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,214 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,215 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6960dd26[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,218 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,218 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,218 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,219 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,219 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,219 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,223 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,223 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,223 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,224 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,224 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,224 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,232 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:42,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,233 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,233 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,333 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:42,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:42,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:42,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:42,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:42,337 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:42,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:42,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:42,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:42,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:42,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:42,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:42,341 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:42,341 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,341 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:42,342 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,355 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/97ed5fa033534535a2a0ae4d11fcfb32 2023-04-13 02:54:42,368 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e85e185fc027412cbeec146662cbe28d 2023-04-13 02:54:42,384 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/ff4c6070446b461683034d7941f2bc4c 2023-04-13 02:54:42,387 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/97ed5fa033534535a2a0ae4d11fcfb32 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32 2023-04-13 02:54:42,392 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32, entries=150, sequenceid=966, filesize=11.9 K 2023-04-13 02:54:42,393 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e85e185fc027412cbeec146662cbe28d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d 2023-04-13 02:54:42,399 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d, entries=150, sequenceid=966, filesize=11.9 K 2023-04-13 02:54:42,400 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/ff4c6070446b461683034d7941f2bc4c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c 2023-04-13 02:54:42,405 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c, entries=150, sequenceid=966, filesize=11.9 K 2023-04-13 02:54:42,406 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for aa0d51e3ad17c60b2f3a2f2437425a6c in 65ms, sequenceid=966, compaction requested=true 2023-04-13 02:54:42,406 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:42,406 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,412 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7ecc3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,419 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,422 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,422 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,425 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:42,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,428 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,428 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,529 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:42,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:42,532 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:42,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:42,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:42,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:42,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:42,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:42,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:42,536 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,536 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:42,536 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:42,536 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,539 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc3a043[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,547 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,547 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,556 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,556 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,656 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:42,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:42,659 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:42,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:42,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:42,660 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,660 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:42,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:42,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:42,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:42,664 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,664 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:42,664 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:42,664 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633b2d14[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,675 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,675 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,677 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,685 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,685 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:42,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:42,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:42,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354482728, max timestamp=1681354482755Num uniques 859; 2023-04-13 02:54:42,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:42,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354482728, max timestamp=1681354482755Num uniques 859; 2023-04-13 02:54:42,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354482728, max timestamp=1681354482755Num uniques 859; 2023-04-13 02:54:42,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:42,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:54:42,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354542770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354542770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354542772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354542773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354542773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,785 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:42,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:42,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:42,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:42,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:42,788 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:42,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:42,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:42,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:42,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:42,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:42,793 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,793 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:42,793 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:42,793 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d45b8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,799 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,802 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,802 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:42,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,811 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,811 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354542874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354542874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354542876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354542877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:42,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354542877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,911 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:42,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:42,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:42,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:42,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:42,915 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,916 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:42,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:42,918 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,919 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:42,919 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:42,919 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:42,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30703104[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:42,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:42,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:42,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,924 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:42,928 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:42,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:42,928 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:42,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:42,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:42,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:42,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:42,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:42,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:42,934 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:42,935 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:42,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,035 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:43,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:43,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:43,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:43,038 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:43,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:43,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:43,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,041 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:43,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:43,042 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:43,042 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,042 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:43,042 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:43,042 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:43,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:43,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:43,044 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31304ef1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:43,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:43,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:43,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:43,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:43,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:43,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:43,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:43,051 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:43,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:43,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:43,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:43,051 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:43,053 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,053 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:43,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:43,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:43,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:43,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:43,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:43,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:43,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:43,058 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:43,058 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:43,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:43,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/bd1a57db61d643b7b9429544d61dcc4d 2023-04-13 02:54:43,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/1876aa97d55a4688954c9a586603397d 2023-04-13 02:54:43,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/78304e7d7170407f9912f745c2e555b4 2023-04-13 02:54:43,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/bd1a57db61d643b7b9429544d61dcc4d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d 2023-04-13 02:54:43,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d, entries=150, sequenceid=978, filesize=11.9 K 2023-04-13 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/1876aa97d55a4688954c9a586603397d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d 2023-04-13 02:54:43,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d, entries=150, sequenceid=978, filesize=11.9 K 2023-04-13 02:54:43,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/78304e7d7170407f9912f745c2e555b4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4 2023-04-13 02:54:43,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4, entries=150, sequenceid=978, filesize=11.9 K 2023-04-13 02:54:43,148 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for aa0d51e3ad17c60b2f3a2f2437425a6c in 450ms, sequenceid=978, compaction requested=true 2023-04-13 02:54:43,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,148 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:43,148 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:43,148 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:43,148 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,148 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,149 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:43,149 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:43,149 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,149 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c86b53ec13e8474797d749705b30464d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=47.7 K 2023-04-13 02:54:43,149 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:43,150 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:43,150 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,150 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/be1bb2ad28414011886dd484294a6032, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=47.7 K 2023-04-13 02:54:43,150 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c86b53ec13e8474797d749705b30464d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:43,150 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting be1bb2ad28414011886dd484294a6032, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:43,150 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d324f7837d4d49f8b177f64aa9d8f596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681354480394 2023-04-13 02:54:43,151 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8073a0fe3edb4e12a1fd75b7d34d516b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681354480394 2023-04-13 02:54:43,151 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 97ed5fa033534535a2a0ae4d11fcfb32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1681354481566 2023-04-13 02:54:43,151 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e85e185fc027412cbeec146662cbe28d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1681354481566 2023-04-13 02:54:43,152 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bd1a57db61d643b7b9429544d61dcc4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,152 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1876aa97d55a4688954c9a586603397d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,158 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:43,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:43,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:43,172 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:43,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:43,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:43,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:43,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,178 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#204 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:43,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:43,183 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#205 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:43,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:43,185 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:43,185 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:43,200 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/e6a0d2f4df6d46ffb3b00784f6116bdc 2023-04-13 02:54:43,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:43,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,598 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/2a364e5a4f9644a08bdc8621c11f406c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/2a364e5a4f9644a08bdc8621c11f406c 2023-04-13 02:54:43,602 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/3cbc7e5434934df7834e900aaedd7262 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/3cbc7e5434934df7834e900aaedd7262 2023-04-13 02:54:43,604 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into 2a364e5a4f9644a08bdc8621c11f406c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:43,604 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,604 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=12, startTime=1681354483148; duration=0sec 2023-04-13 02:54:43,604 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,604 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:43,606 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:43,606 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:43,606 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,606 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e5858e13ca04441c82daa7dc502d6a3c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=47.7 K 2023-04-13 02:54:43,607 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e5858e13ca04441c82daa7dc502d6a3c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1681354479187 2023-04-13 02:54:43,607 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c5c144793df44fcabff01ab9f701e883, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1681354480394 2023-04-13 02:54:43,608 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ff4c6070446b461683034d7941f2bc4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1681354481566 2023-04-13 02:54:43,608 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into 3cbc7e5434934df7834e900aaedd7262(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:43,608 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,608 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=12, startTime=1681354483148; duration=0sec 2023-04-13 02:54:43,608 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 78304e7d7170407f9912f745c2e555b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,608 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:43,616 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/c54583ca627c4e62a50bf0e2545136a2 2023-04-13 02:54:43,617 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#208 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,656 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/8b3c6f112dab41648667e35ac3c96762 2023-04-13 02:54:43,656 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/b1716759c9624b008fa8bba667dd3b63 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/b1716759c9624b008fa8bba667dd3b63 2023-04-13 02:54:43,661 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/e6a0d2f4df6d46ffb3b00784f6116bdc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc 2023-04-13 02:54:43,665 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into b1716759c9624b008fa8bba667dd3b63(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:43,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,665 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=12, startTime=1681354483148; duration=0sec 2023-04-13 02:54:43,665 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:43,667 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-13 02:54:43,668 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/c54583ca627c4e62a50bf0e2545136a2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2 2023-04-13 02:54:43,675 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-13 02:54:43,676 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/8b3c6f112dab41648667e35ac3c96762 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762 2023-04-13 02:54:43,682 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-13 02:54:43,683 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for aa0d51e3ad17c60b2f3a2f2437425a6c in 498ms, sequenceid=1015, compaction requested=false 2023-04-13 02:54:43,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,683 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:43,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:43,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:43,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:43,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,690 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:43,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd9864e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:43,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:43,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,693 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:43,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:43,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:43,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,695 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,695 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:43,695 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:43,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:43,698 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:43,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:43,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:43,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:43,698 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:43,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:43,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:43,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:43,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,702 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:43,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:43,705 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:43,705 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:43,705 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:43,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:43,705 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:43,706 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:43,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:43,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:43,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1031 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/28d835f4c248445aba0fc1d74bb8b757 2023-04-13 02:54:43,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1031 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/dc8787be91f6459e90bc9af9e4a0ae41 2023-04-13 02:54:43,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1031 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/d0315cb67b4140f59c98c39e57c35aa8 2023-04-13 02:54:43,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/28d835f4c248445aba0fc1d74bb8b757 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757 2023-04-13 02:54:43,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757, entries=200, sequenceid=1031, filesize=14.3 K 2023-04-13 02:54:43,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/dc8787be91f6459e90bc9af9e4a0ae41 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41 2023-04-13 02:54:43,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41, entries=150, sequenceid=1031, filesize=11.9 K 2023-04-13 02:54:43,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/d0315cb67b4140f59c98c39e57c35aa8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8 2023-04-13 02:54:43,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8, entries=150, sequenceid=1031, filesize=11.9 K 2023-04-13 02:54:43,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for aa0d51e3ad17c60b2f3a2f2437425a6c in 58ms, sequenceid=1031, compaction requested=true 2023-04-13 02:54:43,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:43,778 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:43,778 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:43,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,779 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:43,779 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:43,779 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:43,780 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,780 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/2a364e5a4f9644a08bdc8621c11f406c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=36.0 K 2023-04-13 02:54:43,779 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:43,780 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,780 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/3cbc7e5434934df7834e900aaedd7262, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=38.4 K 2023-04-13 02:54:43,780 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3cbc7e5434934df7834e900aaedd7262, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,781 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2a364e5a4f9644a08bdc8621c11f406c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,781 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e6a0d2f4df6d46ffb3b00784f6116bdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1015, earliestPutTs=1681354482765 2023-04-13 02:54:43,781 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c54583ca627c4e62a50bf0e2545136a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1015, earliestPutTs=1681354482765 2023-04-13 02:54:43,781 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 28d835f4c248445aba0fc1d74bb8b757, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:43,782 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting dc8787be91f6459e90bc9af9e4a0ae41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:43,790 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#213 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,793 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,806 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:43,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:43,813 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:43,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:43,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:43,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:43,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:43,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:43,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:43,819 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/a7dc5625db1c488b9a5203a804f45795 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a7dc5625db1c488b9a5203a804f45795 2023-04-13 02:54:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:43,825 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into a7dc5625db1c488b9a5203a804f45795(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:43,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:43,825 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=13, startTime=1681354483778; duration=0sec 2023-04-13 02:54:43,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:43,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:54:43,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:54:43,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:43,826 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,826 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/b1716759c9624b008fa8bba667dd3b63, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=36.0 K 2023-04-13 02:54:43,826 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:43,826 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:43,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b1716759c9624b008fa8bba667dd3b63, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1681354482688 2023-04-13 02:54:43,826 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,827 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8b3c6f112dab41648667e35ac3c96762, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1015, earliestPutTs=1681354482765 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:43,827 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:43,827 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d0315cb67b4140f59c98c39e57c35aa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:43,833 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:43,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:43,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:43,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354543951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354543952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354543955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354543955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:43,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:43,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354543955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354544154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354544154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354544157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354544158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354544158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,218 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/7f713fb103a44dd194f4a92081c663b5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/7f713fb103a44dd194f4a92081c663b5 2023-04-13 02:54:44,225 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into 7f713fb103a44dd194f4a92081c663b5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:44,225 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:44,225 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=13, startTime=1681354483778; duration=0sec 2023-04-13 02:54:44,225 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:44,256 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/6f501904ac8142968c1e9cf30b356b8f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/6f501904ac8142968c1e9cf30b356b8f 2023-04-13 02:54:44,260 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into 6f501904ac8142968c1e9cf30b356b8f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:44,260 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:44,260 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=13, startTime=1681354483778; duration=0sec 2023-04-13 02:54:44,260 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:44,262 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1052 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/a00acb4a8fd545c997ab617f8d934e4e 2023-04-13 02:54:44,279 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1052 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/6efc2b9393e347e9a091cbcb97a88a3e 2023-04-13 02:54:44,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354544458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354544458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354544460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354544461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354544461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,689 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1052 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/ba1b97cc86f248b3a6be4bf7a3161508 2023-04-13 02:54:44,694 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/a00acb4a8fd545c997ab617f8d934e4e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e 2023-04-13 02:54:44,698 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e, entries=150, sequenceid=1052, filesize=11.9 K 2023-04-13 02:54:44,698 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/6efc2b9393e347e9a091cbcb97a88a3e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e 2023-04-13 02:54:44,702 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e, entries=150, sequenceid=1052, filesize=11.9 K 2023-04-13 02:54:44,703 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/ba1b97cc86f248b3a6be4bf7a3161508 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508 2023-04-13 02:54:44,706 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508, entries=150, sequenceid=1052, filesize=11.9 K 2023-04-13 02:54:44,707 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for aa0d51e3ad17c60b2f3a2f2437425a6c in 881ms, sequenceid=1052, compaction requested=false 2023-04-13 02:54:44,707 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:44,707 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:44,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,709 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dcf7208[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:44,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:44,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,713 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:44,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:44,715 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:44,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:44,718 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:44,718 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:44,718 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:44,718 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:44,718 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:44,721 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,721 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:44,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,722 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,723 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:44,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:44,726 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,726 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,726 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:44,826 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:44,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:44,835 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:44,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:44,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:44,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:44,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,836 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,837 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:44,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:44,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:44,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:44,840 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:44,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,848 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/852db0574a86436884e0711e8c845a44 2023-04-13 02:54:44,861 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/1bbf03490dad4bd1a3d7111265097fb2 2023-04-13 02:54:44,889 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/0bbd5b6d91f94fa78106eadc5ea1af5c 2023-04-13 02:54:44,894 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/852db0574a86436884e0711e8c845a44 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44 2023-04-13 02:54:44,898 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 02:54:44,899 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/1bbf03490dad4bd1a3d7111265097fb2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2 2023-04-13 02:54:44,905 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 02:54:44,906 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/0bbd5b6d91f94fa78106eadc5ea1af5c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c 2023-04-13 02:54:44,912 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 02:54:44,916 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for aa0d51e3ad17c60b2f3a2f2437425a6c in 76ms, sequenceid=1067, compaction requested=true 2023-04-13 02:54:44,917 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:44,917 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:44,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:44,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:44,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8696a4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:44,921 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:44,921 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:44,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:44,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:44,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:44,929 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:44,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:44,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:44,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:44,929 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:44,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:44,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:44,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:44,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:44,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:44,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:44,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:44,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:44,937 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:44,937 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:44,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:44,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:44,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:44,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:44,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:44,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:44,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:44,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:44,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1080 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/63cd74b317b44f38a7920cea3db940d8 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354484982, max timestamp=1681354484990Num uniques 859; 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354484982, max timestamp=1681354484990Num uniques 859; 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354484982, max timestamp=1681354484990Num uniques 859; 2023-04-13 02:54:44,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:45,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354545021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354545021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354545023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354545022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354545023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,037 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,040 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,045 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,047 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff20c03[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,054 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,054 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,061 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,061 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354545128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354545129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354545129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354545129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354545129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,161 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,164 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,173 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28de0c45[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,174 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,174 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,175 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,177 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,179 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,179 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,181 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,181 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,181 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:45,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,187 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,187 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,287 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,291 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e976464[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,302 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,305 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,305 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,311 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:45,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,311 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,311 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354545331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354545332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354545332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354545332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354545333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1080 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/57c46351c02642ff9bb0adad759ae1ac 2023-04-13 02:54:45,411 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,415 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564ce6a4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,434 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,434 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,439 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,442 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,442 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:45,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,442 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,442 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,542 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,546 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,549 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,549 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,550 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,550 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,551 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a17a7a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,553 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,553 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,559 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,559 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,561 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,562 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,565 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,565 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354545635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354545635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354545636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354545636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354545637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,665 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,669 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c5f72f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:45,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,689 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,690 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,790 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,793 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:45,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,799 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1080 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/5f70381611b74898acaaa378c7bf7287 2023-04-13 02:54:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:45,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f40937[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:45,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:45,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:45,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:45,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:45,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/63cd74b317b44f38a7920cea3db940d8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8 2023-04-13 02:54:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:45,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8, entries=150, sequenceid=1080, filesize=11.9 K 2023-04-13 02:54:45,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/57c46351c02642ff9bb0adad759ae1ac as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac 2023-04-13 02:54:45,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac, entries=150, sequenceid=1080, filesize=11.9 K 2023-04-13 02:54:45,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/5f70381611b74898acaaa378c7bf7287 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287 2023-04-13 02:54:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:45,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:45,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:45,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:45,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:45,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:45,816 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:45,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287, entries=150, sequenceid=1080, filesize=11.9 K 2023-04-13 02:54:45,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=120.76 KB/123660 for aa0d51e3ad17c60b2f3a2f2437425a6c in 851ms, sequenceid=1080, compaction requested=true 2023-04-13 02:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:45,820 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,820 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:45,820 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:45,820 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:45,820 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:45,821 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:45,822 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:45,822 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,822 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:45,822 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:45,822 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,822 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,822 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/7f713fb103a44dd194f4a92081c663b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.0 K 2023-04-13 02:54:45,822 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a7dc5625db1c488b9a5203a804f45795, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.0 K 2023-04-13 02:54:45,823 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7f713fb103a44dd194f4a92081c663b5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:45,824 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a7dc5625db1c488b9a5203a804f45795, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:45,824 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6efc2b9393e347e9a091cbcb97a88a3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1052, earliestPutTs=1681354483732 2023-04-13 02:54:45,824 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1bbf03490dad4bd1a3d7111265097fb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681354483847 2023-04-13 02:54:45,824 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a00acb4a8fd545c997ab617f8d934e4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1052, earliestPutTs=1681354483732 2023-04-13 02:54:45,825 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 57c46351c02642ff9bb0adad759ae1ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:45,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 852db0574a86436884e0711e8c845a44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681354483847 2023-04-13 02:54:45,825 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 63cd74b317b44f38a7920cea3db940d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:45,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:45,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:45,827 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:45,827 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:45,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:45,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:45,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:45,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:45,834 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:45,836 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#226 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:45,927 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:45,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:45,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:45,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:45,932 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:45,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:45,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:45,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:45,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:45,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:45,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:45,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:45,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:45,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:45,935 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-13 02:54:45,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:45,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:45,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:45,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:45,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:45,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:45,946 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1114 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/6d1f64cb9b2d4ebfa2733c115631744d 2023-04-13 02:54:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:46,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:46,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354546147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354546149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354546149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354546150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354546151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354546252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,256 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/f46aa5400cc3458ab3b079659785afb5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f46aa5400cc3458ab3b079659785afb5 2023-04-13 02:54:46,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354546254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354546254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354546254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354546256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,259 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/a4327a8954c340cab7b62972a8a4f610 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/a4327a8954c340cab7b62972a8a4f610 2023-04-13 02:54:46,261 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into f46aa5400cc3458ab3b079659785afb5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:46,262 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:46,262 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=12, startTime=1681354485820; duration=0sec 2023-04-13 02:54:46,262 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:46,262 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:46,263 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:46,263 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:46,263 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:46,263 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/6f501904ac8142968c1e9cf30b356b8f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.0 K 2023-04-13 02:54:46,264 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into a4327a8954c340cab7b62972a8a4f610(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:46,264 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6f501904ac8142968c1e9cf30b356b8f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1031, earliestPutTs=1681354483410 2023-04-13 02:54:46,264 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:46,264 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=12, startTime=1681354485820; duration=0sec 2023-04-13 02:54:46,264 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:46,264 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ba1b97cc86f248b3a6be4bf7a3161508, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1052, earliestPutTs=1681354483732 2023-04-13 02:54:46,264 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0bbd5b6d91f94fa78106eadc5ea1af5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681354483847 2023-04-13 02:54:46,265 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 5f70381611b74898acaaa378c7bf7287, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:46,272 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#229 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:46,356 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1114 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/5d15a1ea0b054876b8a09be856e5b5e4 2023-04-13 02:54:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354546456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354546457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354546458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354546459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354546459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,683 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/ef5ba2385c284c8ab7c933d60b7aae13 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ef5ba2385c284c8ab7c933d60b7aae13 2023-04-13 02:54:46,689 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into ef5ba2385c284c8ab7c933d60b7aae13(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:46,689 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:46,689 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=12, startTime=1681354485821; duration=0sec 2023-04-13 02:54:46,689 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:46,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354546761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354546761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354546762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354546763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354546763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,768 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1114 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/8bb9b03846a64b42b9274d9707382114 2023-04-13 02:54:46,772 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/6d1f64cb9b2d4ebfa2733c115631744d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d 2023-04-13 02:54:46,776 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d, entries=150, sequenceid=1114, filesize=11.9 K 2023-04-13 02:54:46,776 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/5d15a1ea0b054876b8a09be856e5b5e4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4 2023-04-13 02:54:46,780 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4, entries=150, sequenceid=1114, filesize=11.9 K 2023-04-13 02:54:46,781 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/8bb9b03846a64b42b9274d9707382114 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114 2023-04-13 02:54:46,794 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114, entries=150, sequenceid=1114, filesize=11.9 K 2023-04-13 02:54:46,795 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=67.09 KB/68700 for aa0d51e3ad17c60b2f3a2f2437425a6c in 860ms, sequenceid=1114, compaction requested=false 2023-04-13 02:54:46,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:46,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453ca8cb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:46,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:46,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:46,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:46,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:46,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:46,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:46,806 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:46,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:46,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:46,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:46,806 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:46,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:46,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:46,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:46,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:46,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:46,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:46,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:46,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,813 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:46,813 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:46,913 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:46,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:46,917 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:46,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:46,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:46,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:46,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:46,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:46,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:46,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:46,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:46,920 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:46,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:46,928 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1130 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/0a0e3a167600494ead047107eebc8e80 2023-04-13 02:54:46,945 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1130 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/00f50428b67f41a1a99c1be5bbf46f10 2023-04-13 02:54:46,955 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1130 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/d1d40d1d3b97427f93641b06afefa005 2023-04-13 02:54:46,959 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/0a0e3a167600494ead047107eebc8e80 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80 2023-04-13 02:54:46,966 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80, entries=150, sequenceid=1130, filesize=11.9 K 2023-04-13 02:54:46,966 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/00f50428b67f41a1a99c1be5bbf46f10 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10 2023-04-13 02:54:46,970 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10, entries=150, sequenceid=1130, filesize=11.9 K 2023-04-13 02:54:46,970 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-13 02:54:46,971 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/d1d40d1d3b97427f93641b06afefa005 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005 2023-04-13 02:54:46,974 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005, entries=150, sequenceid=1130, filesize=11.9 K 2023-04-13 02:54:46,975 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for aa0d51e3ad17c60b2f3a2f2437425a6c in 55ms, sequenceid=1130, compaction requested=true 2023-04-13 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:46,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,978 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:46,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ec78bc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:46,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:46,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,982 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:46,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:46,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:46,983 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,983 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:46,984 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:46,986 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:46,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:46,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:46,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:46,987 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:46,988 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,988 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:46,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,988 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:46,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:46,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,990 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:46,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:46,993 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:46,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:46,993 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:46,993 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:46,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:46,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,094 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,097 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,098 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,098 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,099 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,099 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,099 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,099 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:47,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,103 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,103 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ce97e1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,106 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,114 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,114 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,117 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,123 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,123 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,223 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,227 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,230 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:47,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d278c5e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,237 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,241 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,241 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,248 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,248 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:47,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:47,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:47,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354547282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354547283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354547284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354547285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354547285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,348 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,353 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,354 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,355 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,355 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,357 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,358 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,358 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dfd1bc1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,362 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,362 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,362 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,364 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,366 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,366 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,368 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,368 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,368 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,369 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,372 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,372 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,372 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,373 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,373 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354547387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354547387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354547388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354547390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354547390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,473 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,477 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,478 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,478 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,481 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,481 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,481 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,481 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,482 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,484 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4950a35a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,487 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,488 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,488 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,489 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,492 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,492 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,492 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,492 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,492 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,499 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,499 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354547591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354547592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354547592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354547593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354547594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,599 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,602 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,607 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,609 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,610 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,610 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,610 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,610 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,613 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,613 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec540e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,616 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,618 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,618 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,620 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,620 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,620 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,620 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,620 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,622 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,622 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,623 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,624 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,627 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,627 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,627 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,627 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,627 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,628 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,628 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1144 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/96195b7b6366441a92f277ec1d2550ea 2023-04-13 02:54:47,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1144 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/8969d65478bb47a59766a671d767f613 2023-04-13 02:54:47,728 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,731 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,735 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,735 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,735 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,735 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,737 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,737 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55786db1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,740 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,741 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,741 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,743 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,746 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,746 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,746 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,746 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,746 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,747 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,747 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,747 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,748 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,751 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,751 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,751 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,752 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,752 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,852 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,856 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,859 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,859 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,859 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,859 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,860 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,862 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe47cc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,865 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,865 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,865 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,866 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,868 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,869 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,871 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,872 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:47,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:47,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,875 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:47,876 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:47,875 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:47,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354547894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354547894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354547895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354547897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354547899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,976 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:47,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:47,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:47,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:47,979 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:47,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:47,979 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:47,980 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,980 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:47,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:47,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:47,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:47,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:47,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:47,983 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,983 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:47,983 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:47,983 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:47,987 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:47,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:47,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf870fa[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:47,990 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:47,991 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,991 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:47,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:47,995 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:47,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:47,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:47,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:47,995 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:47,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:47,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:47,997 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,998 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:48,000 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,000 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:48,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:48,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:48,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:48,001 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:48,001 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:48,101 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:48,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:48,104 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:48,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:48,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:48,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:48,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:48,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,108 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,108 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:48,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:48,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:48,109 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:48,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad21f4f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:48,113 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:48,113 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:48,114 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:48,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:48,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:48,118 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:48,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:48,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:48,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:48,118 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:48,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:48,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1144 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/72adb7854aab42e58ec06c7f62366ec0 2023-04-13 02:54:48,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:48,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/96195b7b6366441a92f277ec1d2550ea as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea 2023-04-13 02:54:48,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:48,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:48,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:48,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,127 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,127 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,127 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:48,127 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:48,127 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:48,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:48,127 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:48,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:48,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea, entries=150, sequenceid=1144, filesize=11.9 K 2023-04-13 02:54:48,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/8969d65478bb47a59766a671d767f613 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613 2023-04-13 02:54:48,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613, entries=150, sequenceid=1144, filesize=11.9 K 2023-04-13 02:54:48,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/72adb7854aab42e58ec06c7f62366ec0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0 2023-04-13 02:54:48,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0, entries=150, sequenceid=1144, filesize=11.9 K 2023-04-13 02:54:48,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for aa0d51e3ad17c60b2f3a2f2437425a6c in 873ms, sequenceid=1144, compaction requested=true 2023-04-13 02:54:48,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:48,143 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:48,143 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:48,143 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:48,145 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:48,145 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:48,145 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:48,145 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:48,145 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,145 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,145 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f46aa5400cc3458ab3b079659785afb5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.1 K 2023-04-13 02:54:48,145 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/a4327a8954c340cab7b62972a8a4f610, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.1 K 2023-04-13 02:54:48,146 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f46aa5400cc3458ab3b079659785afb5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:48,146 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a4327a8954c340cab7b62972a8a4f610, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:48,146 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5d15a1ea0b054876b8a09be856e5b5e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1114, earliestPutTs=1681354484998 2023-04-13 02:54:48,147 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6d1f64cb9b2d4ebfa2733c115631744d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1114, earliestPutTs=1681354484998 2023-04-13 02:54:48,147 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0a0e3a167600494ead047107eebc8e80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1130, earliestPutTs=1681354486145 2023-04-13 02:54:48,147 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 00f50428b67f41a1a99c1be5bbf46f10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1130, earliestPutTs=1681354486145 2023-04-13 02:54:48,148 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 96195b7b6366441a92f277ec1d2550ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:48,148 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8969d65478bb47a59766a671d767f613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:48,157 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:48,161 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#238 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:48,187 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/c67bacc62fcd4ccaa8a18e9575626efd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c67bacc62fcd4ccaa8a18e9575626efd 2023-04-13 02:54:48,194 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into c67bacc62fcd4ccaa8a18e9575626efd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:48,194 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:48,194 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=12, startTime=1681354488143; duration=0sec 2023-04-13 02:54:48,194 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:48,194 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:54:48,195 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:54:48,195 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:48,195 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,196 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ef5ba2385c284c8ab7c933d60b7aae13, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=48.1 K 2023-04-13 02:54:48,196 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ef5ba2385c284c8ab7c933d60b7aae13, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1080, earliestPutTs=1681354484967 2023-04-13 02:54:48,196 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8bb9b03846a64b42b9274d9707382114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1114, earliestPutTs=1681354484998 2023-04-13 02:54:48,197 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d1d40d1d3b97427f93641b06afefa005, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1130, earliestPutTs=1681354486145 2023-04-13 02:54:48,197 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 72adb7854aab42e58ec06c7f62366ec0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:48,205 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:48,215 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/95c16a18826e4ef1b8fbfb57719ccf1a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/95c16a18826e4ef1b8fbfb57719ccf1a 2023-04-13 02:54:48,220 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into 95c16a18826e4ef1b8fbfb57719ccf1a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:48,220 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:48,220 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=12, startTime=1681354488143; duration=0sec 2023-04-13 02:54:48,220 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:48,227 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:48,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:48,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:48,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:48,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:48,231 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:48,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:48,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:48,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:48,232 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,232 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,233 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,233 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,233 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:48,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:48,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:48,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:48,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:48,236 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:48,236 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:48,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:48,249 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1166 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/dc9342c0f2ae4c50be17664158e42bce 2023-04-13 02:54:48,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:48,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354548406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354548406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354548407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354548407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354548410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354548511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354548511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354548511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354548511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354548513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,586 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/74f06316dce543c592f6a9406b3d13ba as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/74f06316dce543c592f6a9406b3d13ba 2023-04-13 02:54:48,590 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into 74f06316dce543c592f6a9406b3d13ba(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:48,590 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:48,590 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=12, startTime=1681354488143; duration=0sec 2023-04-13 02:54:48,590 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:48,662 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1166 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/443a38400ee74d28b1d9dea66b910d59 2023-04-13 02:54:48,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354548714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354548714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354548715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354548716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:48,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:48,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354548717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354549016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354549016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354549018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354549020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354549021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,075 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1166 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/98a901fd5293440382b9cfa3350e700a 2023-04-13 02:54:49,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/dc9342c0f2ae4c50be17664158e42bce as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce 2023-04-13 02:54:49,083 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce, entries=150, sequenceid=1166, filesize=11.9 K 2023-04-13 02:54:49,084 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/443a38400ee74d28b1d9dea66b910d59 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59 2023-04-13 02:54:49,088 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59, entries=150, sequenceid=1166, filesize=11.9 K 2023-04-13 02:54:49,089 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/98a901fd5293440382b9cfa3350e700a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a 2023-04-13 02:54:49,092 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a, entries=150, sequenceid=1166, filesize=11.9 K 2023-04-13 02:54:49,093 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for aa0d51e3ad17c60b2f3a2f2437425a6c in 857ms, sequenceid=1166, compaction requested=false 2023-04-13 02:54:49,093 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:49,093 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:49,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:49,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1e441[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:49,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:49,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,099 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:49,100 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:49,100 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,100 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:49,101 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:49,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:49,115 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:49,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:49,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:49,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:49,115 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:49,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:49,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:49,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:49,117 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:49,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:49,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:49,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:49,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:49,121 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:49,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:49,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:49,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:49,121 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:49,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,221 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:49,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:49,228 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:49,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:49,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:49,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:49,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:49,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,231 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:49,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:49,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:49,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:49,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:49,233 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:49,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:49,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:49,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354549534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354549537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354549537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354549538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354549537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354549639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,640 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1179 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/37fc85230eeb4c06ab3f010c6f53f06c 2023-04-13 02:54:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354549641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354549642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354549642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354549642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354549842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354549844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354549844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354549845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354549845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,053 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1179 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/65fa05bbdfbb4f2f873253a9111f6bd7 2023-04-13 02:54:50,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354550144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354550149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354550149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354550150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354550150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,464 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1179 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/540069a287cc4f2db22cd8821ffb8476 2023-04-13 02:54:50,468 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/37fc85230eeb4c06ab3f010c6f53f06c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c 2023-04-13 02:54:50,472 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c, entries=150, sequenceid=1179, filesize=11.9 K 2023-04-13 02:54:50,473 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/65fa05bbdfbb4f2f873253a9111f6bd7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7 2023-04-13 02:54:50,476 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7, entries=150, sequenceid=1179, filesize=11.9 K 2023-04-13 02:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/540069a287cc4f2db22cd8821ffb8476 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476 2023-04-13 02:54:50,480 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476, entries=150, sequenceid=1179, filesize=11.9 K 2023-04-13 02:54:50,481 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for aa0d51e3ad17c60b2f3a2f2437425a6c in 1248ms, sequenceid=1179, compaction requested=true 2023-04-13 02:54:50,481 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:50,481 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:50,483 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e930a30[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:50,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:50,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,486 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,486 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:50,486 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:50,486 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:50,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:50,491 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:50,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:50,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:50,493 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:50,493 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:50,493 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:50,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:50,494 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:50,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:50,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:50,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:50,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:50,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:50,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:50,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:50,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:50,500 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:50,500 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:50,600 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:50,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:50,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:50,603 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:50,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:50,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:50,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:50,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:50,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:50,606 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:50,606 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,606 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:50,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:50,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:50,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:50,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:50,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:50,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:50,608 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:50,608 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:50,621 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1199 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/688d3979a95e441f989917650125582d 2023-04-13 02:54:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:50,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354550659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354550660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354550661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354550661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354550661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354550764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354550765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354550765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354550765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354550766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354550967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354550968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354550968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354550968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:50,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354550969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,035 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1199 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/0d4484ab7643436e8d84c780a9243e9b 2023-04-13 02:54:51,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354551269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354551271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354551271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354551272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354551272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,446 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1199 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/667138aa700f4389b6e44cda011ccb58 2023-04-13 02:54:51,451 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/688d3979a95e441f989917650125582d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d 2023-04-13 02:54:51,454 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d, entries=150, sequenceid=1199, filesize=11.9 K 2023-04-13 02:54:51,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/0d4484ab7643436e8d84c780a9243e9b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b 2023-04-13 02:54:51,459 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b, entries=150, sequenceid=1199, filesize=11.9 K 2023-04-13 02:54:51,459 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/667138aa700f4389b6e44cda011ccb58 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58 2023-04-13 02:54:51,462 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58, entries=150, sequenceid=1199, filesize=11.9 K 2023-04-13 02:54:51,463 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for aa0d51e3ad17c60b2f3a2f2437425a6c in 855ms, sequenceid=1199, compaction requested=true 2023-04-13 02:54:51,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:51,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:51,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:51,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:51,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:51,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,465 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3113ccb8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:51,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:51,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:51,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:51,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:51,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:51,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:51,473 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:51,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:51,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:51,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:51,473 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:51,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:51,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:51,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:51,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:51,480 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:51,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:51,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:51,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:51,481 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:51,481 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:51,581 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:51,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:51,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:51,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:51,584 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:51,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:51,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:51,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:51,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:51,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:51,586 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:51,586 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,586 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:51,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:51,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:51,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:51,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:51,587 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:51,587 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:51,587 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:51,588 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:51,595 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1211 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/015ff34a995140e288a245c86ce117a8 2023-04-13 02:54:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:51,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:51,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354551789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354551790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354551790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354551793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354551794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354551896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354551896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354551896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354551897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:51,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:51,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354551898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,029 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1211 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e1c079b33bb74d25ac0c9c0276cd6024 2023-04-13 02:54:52,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354552098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354552099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354552099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354552100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354552100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354552402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354552402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354552403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354552406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,439 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1211 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/9ae9941110504c0aad9882275e218796 2023-04-13 02:54:52,443 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/015ff34a995140e288a245c86ce117a8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8 2023-04-13 02:54:52,447 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8, entries=150, sequenceid=1211, filesize=11.9 K 2023-04-13 02:54:52,447 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e1c079b33bb74d25ac0c9c0276cd6024 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024 2023-04-13 02:54:52,451 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024, entries=150, sequenceid=1211, filesize=11.9 K 2023-04-13 02:54:52,452 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/9ae9941110504c0aad9882275e218796 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796 2023-04-13 02:54:52,455 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796, entries=150, sequenceid=1211, filesize=11.9 K 2023-04-13 02:54:52,456 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for aa0d51e3ad17c60b2f3a2f2437425a6c in 869ms, sequenceid=1211, compaction requested=true 2023-04-13 02:54:52,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:52,456 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:52,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:52,458 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf185b2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:52,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:52,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,462 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:52,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:52,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:52,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:52,463 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:52,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:52,467 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:52,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:52,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:52,467 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:52,467 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:52,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:52,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:52,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:52,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:52,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:52,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:52,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:52,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:52,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:52,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:52,473 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:52,473 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:52,573 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:52,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:52,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:52,577 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:52,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:52,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:52,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:52,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:52,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:52,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:52,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:52,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:52,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:52,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:52,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:52,582 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:52,583 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:52,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:52,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:52,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354552915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354552915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354552918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354552919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:52,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354552919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,001 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1231 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/740b112100004717ac21fd4eb6633109 2023-04-13 02:54:53,011 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1231 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/da7f2daff6a9489ab631ccd464bcbd56 2023-04-13 02:54:53,021 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1231 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/a97445e89d3349dcb2da4d5bbe2771be 2023-04-13 02:54:53,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354553021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354553021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,025 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/740b112100004717ac21fd4eb6633109 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109 2023-04-13 02:54:53,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354553022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354553024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354553024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,029 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109, entries=150, sequenceid=1231, filesize=11.9 K 2023-04-13 02:54:53,029 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/da7f2daff6a9489ab631ccd464bcbd56 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56 2023-04-13 02:54:53,033 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56, entries=150, sequenceid=1231, filesize=11.9 K 2023-04-13 02:54:53,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/a97445e89d3349dcb2da4d5bbe2771be as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be 2023-04-13 02:54:53,037 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be, entries=150, sequenceid=1231, filesize=11.9 K 2023-04-13 02:54:53,038 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for aa0d51e3ad17c60b2f3a2f2437425a6c in 455ms, sequenceid=1231, compaction requested=true 2023-04-13 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:53,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:53,040 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c18561[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:53,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:53,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,044 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:53,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:53,046 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:53,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:53,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:53,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:53,048 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:53,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:53,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:53,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:53,048 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:53,053 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,053 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,053 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:53,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:53,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:53,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:53,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:53,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:53,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:53,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:53,058 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:53,058 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:53,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:53,158 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:53,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:53,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:53,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:53,162 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:53,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:53,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:53,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:53,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:53,164 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:53,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:53,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:53,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:53,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:53,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:53,166 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:53,166 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:53,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:53,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354553240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354553243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354553244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354553244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354553245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354553345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354553347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354553348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354553348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354553349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354553550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354553550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354553551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354553552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354553553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,574 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1244 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/bc2607d82b1e4b1f9d441a9342a42a67 2023-04-13 02:54:53,584 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1244 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/28b0499bd3f94b0ca9e99b51dc034435 2023-04-13 02:54:53,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354553853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354553854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354553854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354553855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:53,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354553856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:53,994 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1244 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/024fccfe148e4fa48d73a5aa513f6a6f 2023-04-13 02:54:54,002 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/bc2607d82b1e4b1f9d441a9342a42a67 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67 2023-04-13 02:54:54,006 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67, entries=150, sequenceid=1244, filesize=11.9 K 2023-04-13 02:54:54,007 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/28b0499bd3f94b0ca9e99b51dc034435 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435 2023-04-13 02:54:54,011 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435, entries=150, sequenceid=1244, filesize=11.9 K 2023-04-13 02:54:54,011 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/024fccfe148e4fa48d73a5aa513f6a6f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f 2023-04-13 02:54:54,014 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f, entries=150, sequenceid=1244, filesize=11.9 K 2023-04-13 02:54:54,015 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for aa0d51e3ad17c60b2f3a2f2437425a6c in 849ms, sequenceid=1244, compaction requested=true 2023-04-13 02:54:54,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:54,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:54,016 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:54,018 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:54,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e55cac3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:54,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:54,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,021 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:54,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:54,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:54,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:54,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:54,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:54,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:54,025 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:54,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:54,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:54,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:54,025 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:54,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:54,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:54,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:54,028 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:54,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:54,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:54,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:54,032 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:54,032 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:54,133 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:54,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:54,136 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:54,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:54,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:54,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:54,137 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,137 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:54,138 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,138 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:54,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:54,138 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,138 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:54,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:54,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:54,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:54,139 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:54,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:54,140 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:54,140 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:54,141 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:54,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354554365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354554366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354554366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354554367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354554368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354554469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354554470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354554470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354554471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354554472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,550 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1264 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/a973890444e8437d9bbdaa84a991f347 2023-04-13 02:54:54,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354554672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354554672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354554673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354554675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354554675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,960 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1264 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/4ed1f37109634ed0a7d14283fc0b85af 2023-04-13 02:54:54,970 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1264 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/48d5e18003934a1ab213d2636a3895e2 2023-04-13 02:54:54,974 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/a973890444e8437d9bbdaa84a991f347 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347 2023-04-13 02:54:54,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354554975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,979 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347, entries=150, sequenceid=1264, filesize=11.9 K 2023-04-13 02:54:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,980 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/4ed1f37109634ed0a7d14283fc0b85af as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af 2023-04-13 02:54:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354554977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354554977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354554980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,984 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af, entries=150, sequenceid=1264, filesize=11.9 K 2023-04-13 02:54:54,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354554981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,985 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/48d5e18003934a1ab213d2636a3895e2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2 2023-04-13 02:54:54,988 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2, entries=150, sequenceid=1264, filesize=11.9 K 2023-04-13 02:54:54,989 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for aa0d51e3ad17c60b2f3a2f2437425a6c in 849ms, sequenceid=1264, compaction requested=true 2023-04-13 02:54:54,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:54,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:54,991 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418cbdbd[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:54,993 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:54,993 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,994 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:54,994 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,994 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:54,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:54,999 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:54,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:54,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:54,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:54,999 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,002 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,003 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,005 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,005 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,006 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,006 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,006 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,006 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,106 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,110 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,113 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:54:55,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,123 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1276 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/9fb66be9e77c43dea79c0bd29a5b6f22 2023-04-13 02:54:55,132 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1276 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/6f5ff7a4a8844e078777c39a999280ef 2023-04-13 02:54:55,142 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1276 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/538bbb756e2b43f4aca09cb488add66a 2023-04-13 02:54:55,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/9fb66be9e77c43dea79c0bd29a5b6f22 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22 2023-04-13 02:54:55,152 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22, entries=150, sequenceid=1276, filesize=11.9 K 2023-04-13 02:54:55,153 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/6f5ff7a4a8844e078777c39a999280ef as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef 2023-04-13 02:54:55,157 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef, entries=150, sequenceid=1276, filesize=11.9 K 2023-04-13 02:54:55,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/538bbb756e2b43f4aca09cb488add66a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a 2023-04-13 02:54:55,161 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a, entries=150, sequenceid=1276, filesize=11.9 K 2023-04-13 02:54:55,161 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for aa0d51e3ad17c60b2f3a2f2437425a6c in 48ms, sequenceid=1276, compaction requested=true 2023-04-13 02:54:55,162 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:55,162 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,165 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36276c2b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,172 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,172 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,182 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,182 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,182 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,183 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,183 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,183 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,283 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,287 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,287 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,288 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,288 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,290 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,290 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:55,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,292 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f91d03[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,302 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,304 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,305 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,311 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,312 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,312 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,412 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,415 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:55,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423adc4d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,427 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,427 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,427 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,431 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,436 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,436 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:55,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:55,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354555523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354555523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354555524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354495521, max timestamp=1681354495528Num uniques 859; 2023-04-13 02:54:55,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:55,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354495521, max timestamp=1681354495528Num uniques 859; 2023-04-13 02:54:55,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:55,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354495521, max timestamp=1681354495528Num uniques 859; 2023-04-13 02:54:55,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:54:55,536 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,539 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:55,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ffe0e2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,552 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,554 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,554 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,560 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,560 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354555563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354555567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354555630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354555630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354555630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,660 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,664 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:55,670 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b348e22[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354555671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354555671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,684 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,684 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,690 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,690 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,790 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,793 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:55,797 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216f9742[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,807 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,807 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,815 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,815 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354555834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354555835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354555835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354555876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354555877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/c8c84682b23e4181a5f81caf49d5b850 2023-04-13 02:54:55,915 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:55,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:55,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:55,918 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:55,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:55,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:55,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,920 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:55,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:55,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:55,922 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:55,922 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:55,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,923 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3523ff96[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:55,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:55,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:55,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:55,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:55,931 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:55,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:55,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:55,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:55,931 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:55,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:55,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:55,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:55,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:55,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:55,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:55,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:55,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:55,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:55,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:55,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:55,937 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:55,937 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:55,937 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,037 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:56,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:56,040 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:56,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:56,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:56,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:56,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:56,044 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,044 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:56,044 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:56,044 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:56,051 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc2e120[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:56,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:56,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:56,058 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:56,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:56,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:56,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:56,058 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:56,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,061 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:56,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:56,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,065 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:56,066 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:56,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354556137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354556138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354556141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,166 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:56,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:56,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:56,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:56,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:56,169 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:56,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:56,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:56,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:56,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:56,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:56,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:56,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63482f04[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:56,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:56,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,179 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:56,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,179 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:56,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:56,183 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:56,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:56,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:56,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:56,183 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:56,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354556180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354556180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:56,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:56,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,189 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:56,189 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:56,289 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:56,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:56,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:56,292 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:56,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:56,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:56,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,295 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:56,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:56,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:56,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:56,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:56,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:56,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:56,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec071a8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:56,300 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:56,300 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:56,302 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,302 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:56,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:56,305 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:56,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:56,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:56,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:56,305 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:56,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:56,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:56,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:56,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:56,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:56,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:56,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:56,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:56,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/b9dc0d9f7b18469d82fe2b484a70996c 2023-04-13 02:54:56,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:56,312 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,312 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:56,312 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:56,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:56,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:56,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/e8d6dedeeeec4b3c9df6f011759f5e22 2023-04-13 02:54:56,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/c8c84682b23e4181a5f81caf49d5b850 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850 2023-04-13 02:54:56,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850, entries=150, sequenceid=1289, filesize=11.9 K 2023-04-13 02:54:56,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/b9dc0d9f7b18469d82fe2b484a70996c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c 2023-04-13 02:54:56,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c, entries=150, sequenceid=1289, filesize=11.9 K 2023-04-13 02:54:56,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/e8d6dedeeeec4b3c9df6f011759f5e22 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22 2023-04-13 02:54:56,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22, entries=150, sequenceid=1289, filesize=11.9 K 2023-04-13 02:54:56,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for aa0d51e3ad17c60b2f3a2f2437425a6c in 846ms, sequenceid=1289, compaction requested=true 2023-04-13 02:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:56,339 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-13 02:54:56,339 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-13 02:54:56,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:56,342 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122508 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-13 02:54:56,342 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/A is initiating minor compaction (all files) 2023-04-13 02:54:56,342 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122508 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-13 02:54:56,342 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/A in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,342 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/B is initiating minor compaction (all files) 2023-04-13 02:54:56,342 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/B in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,342 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/74f06316dce543c592f6a9406b3d13ba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=119.6 K 2023-04-13 02:54:56,342 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c67bacc62fcd4ccaa8a18e9575626efd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=119.6 K 2023-04-13 02:54:56,343 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 74f06316dce543c592f6a9406b3d13ba, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:56,343 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c67bacc62fcd4ccaa8a18e9575626efd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:56,344 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dc9342c0f2ae4c50be17664158e42bce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1166, earliestPutTs=1681354487280 2023-04-13 02:54:56,344 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 443a38400ee74d28b1d9dea66b910d59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1166, earliestPutTs=1681354487280 2023-04-13 02:54:56,344 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 37fc85230eeb4c06ab3f010c6f53f06c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1179, earliestPutTs=1681354488405 2023-04-13 02:54:56,344 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 65fa05bbdfbb4f2f873253a9111f6bd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1179, earliestPutTs=1681354488405 2023-04-13 02:54:56,344 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 688d3979a95e441f989917650125582d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1199, earliestPutTs=1681354489530 2023-04-13 02:54:56,345 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 015ff34a995140e288a245c86ce117a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681354490658 2023-04-13 02:54:56,345 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0d4484ab7643436e8d84c780a9243e9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1199, earliestPutTs=1681354489530 2023-04-13 02:54:56,345 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 740b112100004717ac21fd4eb6633109, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1231, earliestPutTs=1681354491789 2023-04-13 02:54:56,345 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e1c079b33bb74d25ac0c9c0276cd6024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681354490658 2023-04-13 02:54:56,345 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bc2607d82b1e4b1f9d441a9342a42a67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1244, earliestPutTs=1681354492914 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting da7f2daff6a9489ab631ccd464bcbd56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1231, earliestPutTs=1681354491789 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a973890444e8437d9bbdaa84a991f347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1264, earliestPutTs=1681354493242 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 28b0499bd3f94b0ca9e99b51dc034435, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1244, earliestPutTs=1681354492914 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9fb66be9e77c43dea79c0bd29a5b6f22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1276, earliestPutTs=1681354494361 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4ed1f37109634ed0a7d14283fc0b85af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1264, earliestPutTs=1681354493242 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c8c84682b23e4181a5f81caf49d5b850, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681354495491 2023-04-13 02:54:56,346 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6f5ff7a4a8844e078777c39a999280ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1276, earliestPutTs=1681354494361 2023-04-13 02:54:56,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b9dc0d9f7b18469d82fe2b484a70996c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681354495491 2023-04-13 02:54:56,362 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#A#compaction#267 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:56,363 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#B#compaction#268 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:56,413 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:56,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:56,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:54:56,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:54:56,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:54:56,416 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:54:56,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:54:56,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:54:56,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:56,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:56,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:54:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:54:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:56,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:54:56,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:54:56,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:54:56,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:54:56,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:54:56,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:54:56,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:54:56,420 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,420 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. started... 2023-04-13 02:54:56,420 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:54:56,420 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:56,421 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:56,421 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:56,421 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:56,421 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:56,421 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:54:56,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:56,649 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. as already flushing 2023-04-13 02:54:56,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354556656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354556659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354556661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55322 deadline: 1681354556685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55340 deadline: 1681354556685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354556761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354556764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354556765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,772 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/b38219c204624221b6328c425ab683f5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/b38219c204624221b6328c425ab683f5 2023-04-13 02:54:56,772 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/c2046e84ccb94919ae5d822105afb7f0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c2046e84ccb94919ae5d822105afb7f0 2023-04-13 02:54:56,777 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/A of aa0d51e3ad17c60b2f3a2f2437425a6c into b38219c204624221b6328c425ab683f5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:56,777 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/B of aa0d51e3ad17c60b2f3a2f2437425a6c into c2046e84ccb94919ae5d822105afb7f0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:56,777 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:56,777 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:56,777 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/A, priority=6, startTime=1681354496339; duration=0sec 2023-04-13 02:54:56,777 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/B, priority=6, startTime=1681354496339; duration=0sec 2023-04-13 02:54:56,777 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:56,777 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:56,777 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-13 02:54:56,780 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122508 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-13 02:54:56,780 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): aa0d51e3ad17c60b2f3a2f2437425a6c/C is initiating minor compaction (all files) 2023-04-13 02:54:56,780 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aa0d51e3ad17c60b2f3a2f2437425a6c/C in TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:56,780 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/95c16a18826e4ef1b8fbfb57719ccf1a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp, totalSize=119.6 K 2023-04-13 02:54:56,781 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 95c16a18826e4ef1b8fbfb57719ccf1a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1144, earliestPutTs=1681354487269 2023-04-13 02:54:56,781 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 98a901fd5293440382b9cfa3350e700a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1166, earliestPutTs=1681354487280 2023-04-13 02:54:56,781 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 540069a287cc4f2db22cd8821ffb8476, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1179, earliestPutTs=1681354488405 2023-04-13 02:54:56,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 667138aa700f4389b6e44cda011ccb58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1199, earliestPutTs=1681354489530 2023-04-13 02:54:56,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9ae9941110504c0aad9882275e218796, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1211, earliestPutTs=1681354490658 2023-04-13 02:54:56,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a97445e89d3349dcb2da4d5bbe2771be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1231, earliestPutTs=1681354491789 2023-04-13 02:54:56,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 024fccfe148e4fa48d73a5aa513f6a6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1244, earliestPutTs=1681354492914 2023-04-13 02:54:56,782 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 48d5e18003934a1ab213d2636a3895e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1264, earliestPutTs=1681354493242 2023-04-13 02:54:56,783 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 538bbb756e2b43f4aca09cb488add66a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1276, earliestPutTs=1681354494361 2023-04-13 02:54:56,783 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e8d6dedeeeec4b3c9df6f011759f5e22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681354495491 2023-04-13 02:54:56,796 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): aa0d51e3ad17c60b2f3a2f2437425a6c#C#compaction#270 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:54:56,803 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/e91325764a604909976727e60419adba as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e91325764a604909976727e60419adba 2023-04-13 02:54:56,806 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in aa0d51e3ad17c60b2f3a2f2437425a6c/C of aa0d51e3ad17c60b2f3a2f2437425a6c into e91325764a604909976727e60419adba(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:54:56,807 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:56,807 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c., storeName=aa0d51e3ad17c60b2f3a2f2437425a6c/C, priority=6, startTime=1681354496339; duration=0sec 2023-04-13 02:54:56,807 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:56,828 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/485b19dfb733455f94fc682e3e23bf88 2023-04-13 02:54:56,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354556965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354556969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:56,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354556970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,241 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e90dc946701c4a2b81fa6590e42bd28f 2023-04-13 02:54:57,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55284 deadline: 1681354557269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55344 deadline: 1681354557274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:54:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55330 deadline: 1681354557274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,592 DEBUG [Thread-1504] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x451c882f to 127.0.0.1:62948 2023-04-13 02:54:57,592 DEBUG [Thread-1500] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06bbe763 to 127.0.0.1:62948 2023-04-13 02:54:57,593 DEBUG [Thread-1504] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,593 DEBUG [Thread-1500] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,593 DEBUG [Thread-1498] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e067bfc to 127.0.0.1:62948 2023-04-13 02:54:57,593 DEBUG [Thread-1498] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,595 DEBUG [Thread-1502] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c8736d4 to 127.0.0.1:62948 2023-04-13 02:54:57,595 DEBUG [Thread-1496] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x270a3c0c to 127.0.0.1:62948 2023-04-13 02:54:57,595 DEBUG [Thread-1502] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,595 DEBUG [Thread-1496] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,652 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1324 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/844afd0a0c3e4e3bbac67381f35a6c43 2023-04-13 02:54:57,656 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/485b19dfb733455f94fc682e3e23bf88 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/485b19dfb733455f94fc682e3e23bf88 2023-04-13 02:54:57,659 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/485b19dfb733455f94fc682e3e23bf88, entries=150, sequenceid=1324, filesize=11.9 K 2023-04-13 02:54:57,660 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/e90dc946701c4a2b81fa6590e42bd28f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e90dc946701c4a2b81fa6590e42bd28f 2023-04-13 02:54:57,663 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e90dc946701c4a2b81fa6590e42bd28f, entries=150, sequenceid=1324, filesize=11.9 K 2023-04-13 02:54:57,664 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/844afd0a0c3e4e3bbac67381f35a6c43 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/844afd0a0c3e4e3bbac67381f35a6c43 2023-04-13 02:54:57,667 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/844afd0a0c3e4e3bbac67381f35a6c43, entries=150, sequenceid=1324, filesize=11.9 K 2023-04-13 02:54:57,667 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for aa0d51e3ad17c60b2f3a2f2437425a6c in 1247ms, sequenceid=1324, compaction requested=false 2023-04-13 02:54:57,667 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:57,667 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:54:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:54:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b5e6a9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:54:57,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:54:57,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,677 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:54:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:54:57,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:54:57,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:57,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:54:57,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:54:57,681 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:54:57,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:54:57,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:54:57,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:54:57,681 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:54:57,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:54:57,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:57,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:57,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:54:57,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:54:57,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:54:57,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:54:57,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:54:57,689 DEBUG [Thread-1495] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:54:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:54:57,689 DEBUG [Thread-1495] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:54:57,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:57,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:54:57,697 DEBUG [Thread-1491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c6d5986 to 127.0.0.1:62948 2023-04-13 02:54:57,697 DEBUG [Thread-1493] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31182447 to 127.0.0.1:62948 2023-04-13 02:54:57,697 DEBUG [Thread-1491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:57,697 DEBUG [Thread-1493] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:57,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:57,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:57,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:57,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:57,774 DEBUG [Thread-1489] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x025522dc to 127.0.0.1:62948 2023-04-13 02:54:57,774 DEBUG [Thread-1489] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,779 DEBUG [Thread-1485] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b40c526 to 127.0.0.1:62948 2023-04-13 02:54:57,779 DEBUG [Thread-1485] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,780 DEBUG [Thread-1487] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47ae15dc to 127.0.0.1:62948 2023-04-13 02:54:57,780 DEBUG [Thread-1487] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,789 DEBUG [Thread-1495] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:54:57,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 239 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 272 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 234 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 245 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 250 2023-04-13 02:54:57,790 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5191 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5128 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5122 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5186 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 5139 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:54:57,791 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b4a16d4 to 127.0.0.1:62948 2023-04-13 02:54:57,791 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:54:57,791 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:54:57,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:54:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:54:57,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:54:57,795 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354497795"}]},"ts":"1681354497795"} 2023-04-13 02:54:57,796 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:54:57,798 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:54:57,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, UNASSIGN}] 2023-04-13 02:54:57,800 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, UNASSIGN 2023-04-13 02:54:57,800 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=aa0d51e3ad17c60b2f3a2f2437425a6c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:54:57,800 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354497800"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354497800"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354497800"}]},"ts":"1681354497800"} 2023-04-13 02:54:57,801 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:54:57,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:54:57,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:54:57,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing aa0d51e3ad17c60b2f3a2f2437425a6c, disabling compactions & flushes 2023-04-13 02:54:57,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:58,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:54:58,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1340 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/f8012badf6c84491ad9155b23f07b987 2023-04-13 02:54:58,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:54:58,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1340 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/4c18ee6eaa224d369d0b7db3dcb4df92 2023-04-13 02:54:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:54:58,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1340 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/00b81cc287df4bf89376a172604ad42a 2023-04-13 02:54:58,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/f8012badf6c84491ad9155b23f07b987 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f8012badf6c84491ad9155b23f07b987 2023-04-13 02:54:58,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f8012badf6c84491ad9155b23f07b987, entries=150, sequenceid=1340, filesize=11.9 K 2023-04-13 02:54:58,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/4c18ee6eaa224d369d0b7db3dcb4df92 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4c18ee6eaa224d369d0b7db3dcb4df92 2023-04-13 02:54:58,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4c18ee6eaa224d369d0b7db3dcb4df92, entries=150, sequenceid=1340, filesize=11.9 K 2023-04-13 02:54:58,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/00b81cc287df4bf89376a172604ad42a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/00b81cc287df4bf89376a172604ad42a 2023-04-13 02:54:58,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/00b81cc287df4bf89376a172604ad42a, entries=150, sequenceid=1340, filesize=11.9 K 2023-04-13 02:54:58,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for aa0d51e3ad17c60b2f3a2f2437425a6c in 1240ms, sequenceid=1340, compaction requested=true 2023-04-13 02:54:58,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:54:58,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:58,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:58,937 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. because compaction request was cancelled 2023-04-13 02:54:58,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:58,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. after waiting 0 ms 2023-04-13 02:54:58,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:54:58,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:54:58,938 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. because compaction request was cancelled 2023-04-13 02:54:58,937 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. because compaction request was cancelled 2023-04-13 02:54:58,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing aa0d51e3ad17c60b2f3a2f2437425a6c 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-13 02:54:58,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=A 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=B 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aa0d51e3ad17c60b2f3a2f2437425a6c, store=C 2023-04-13 02:54:58,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:54:59,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1346 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/ae6ddd4aa17745c7bc5dd13f069fdd54 2023-04-13 02:54:59,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1346 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/bb0f45e3d1124ece9b95fe34c0327232 2023-04-13 02:54:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:55:00,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1346 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/9905151273a64969a671e79101fc5911 2023-04-13 02:55:00,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/A/ae6ddd4aa17745c7bc5dd13f069fdd54 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/ae6ddd4aa17745c7bc5dd13f069fdd54 2023-04-13 02:55:00,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/ae6ddd4aa17745c7bc5dd13f069fdd54, entries=50, sequenceid=1346, filesize=7.1 K 2023-04-13 02:55:00,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/B/bb0f45e3d1124ece9b95fe34c0327232 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bb0f45e3d1124ece9b95fe34c0327232 2023-04-13 02:55:00,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bb0f45e3d1124ece9b95fe34c0327232, entries=50, sequenceid=1346, filesize=7.1 K 2023-04-13 02:55:00,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/.tmp/C/9905151273a64969a671e79101fc5911 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9905151273a64969a671e79101fc5911 2023-04-13 02:55:00,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9905151273a64969a671e79101fc5911, entries=50, sequenceid=1346, filesize=7.1 K 2023-04-13 02:55:00,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for aa0d51e3ad17c60b2f3a2f2437425a6c in 1241ms, sequenceid=1346, compaction requested=true 2023-04-13 02:55:00,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c86b53ec13e8474797d749705b30464d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/3cbc7e5434934df7834e900aaedd7262, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a7dc5625db1c488b9a5203a804f45795, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f46aa5400cc3458ab3b079659785afb5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/74f06316dce543c592f6a9406b3d13ba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850] to archive 2023-04-13 02:55:00,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:00,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/4b72540bb7624f70a136596096fd7b1b 2023-04-13 02:55:00,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f386d51c655c43a6b1d74deb349e8b17 2023-04-13 02:55:00,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/21ca1cf8dc584776a5a7f361b64adb80 2023-04-13 02:55:00,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/95bbe55a8ca94663b0e4c599f4e579f9 2023-04-13 02:55:00,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c86b53ec13e8474797d749705b30464d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c86b53ec13e8474797d749705b30464d 2023-04-13 02:55:00,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/d324f7837d4d49f8b177f64aa9d8f596 2023-04-13 02:55:00,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/97ed5fa033534535a2a0ae4d11fcfb32 2023-04-13 02:55:00,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/3cbc7e5434934df7834e900aaedd7262 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/3cbc7e5434934df7834e900aaedd7262 2023-04-13 02:55:00,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bd1a57db61d643b7b9429544d61dcc4d 2023-04-13 02:55:00,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/e6a0d2f4df6d46ffb3b00784f6116bdc 2023-04-13 02:55:00,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/28d835f4c248445aba0fc1d74bb8b757 2023-04-13 02:55:00,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a7dc5625db1c488b9a5203a804f45795 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a7dc5625db1c488b9a5203a804f45795 2023-04-13 02:55:00,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a00acb4a8fd545c997ab617f8d934e4e 2023-04-13 02:55:00,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/852db0574a86436884e0711e8c845a44 2023-04-13 02:55:00,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f46aa5400cc3458ab3b079659785afb5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f46aa5400cc3458ab3b079659785afb5 2023-04-13 02:55:00,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/63cd74b317b44f38a7920cea3db940d8 2023-04-13 02:55:00,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/6d1f64cb9b2d4ebfa2733c115631744d 2023-04-13 02:55:00,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/0a0e3a167600494ead047107eebc8e80 2023-04-13 02:55:00,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/74f06316dce543c592f6a9406b3d13ba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/74f06316dce543c592f6a9406b3d13ba 2023-04-13 02:55:00,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/96195b7b6366441a92f277ec1d2550ea 2023-04-13 02:55:00,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/dc9342c0f2ae4c50be17664158e42bce 2023-04-13 02:55:00,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/37fc85230eeb4c06ab3f010c6f53f06c 2023-04-13 02:55:00,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/688d3979a95e441f989917650125582d 2023-04-13 02:55:00,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/015ff34a995140e288a245c86ce117a8 2023-04-13 02:55:00,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/740b112100004717ac21fd4eb6633109 2023-04-13 02:55:00,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/bc2607d82b1e4b1f9d441a9342a42a67 2023-04-13 02:55:00,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/a973890444e8437d9bbdaa84a991f347 2023-04-13 02:55:00,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/9fb66be9e77c43dea79c0bd29a5b6f22 2023-04-13 02:55:00,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/c8c84682b23e4181a5f81caf49d5b850 2023-04-13 02:55:00,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/be1bb2ad28414011886dd484294a6032, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/2a364e5a4f9644a08bdc8621c11f406c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/7f713fb103a44dd194f4a92081c663b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/a4327a8954c340cab7b62972a8a4f610, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c67bacc62fcd4ccaa8a18e9575626efd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c] to archive 2023-04-13 02:55:00,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:00,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fe3a7203629b4bbca2f2ba45dcb3db15 2023-04-13 02:55:00,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/fb1d1d68139147b6a21787fb7ab867b2 2023-04-13 02:55:00,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b4ee9ce07aa94ed6a7476c1ceb781aaf 2023-04-13 02:55:00,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/be1bb2ad28414011886dd484294a6032 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/be1bb2ad28414011886dd484294a6032 2023-04-13 02:55:00,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bd3ad1e3e41a42be8581a489e92ff94c 2023-04-13 02:55:00,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8073a0fe3edb4e12a1fd75b7d34d516b 2023-04-13 02:55:00,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e85e185fc027412cbeec146662cbe28d 2023-04-13 02:55:00,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/2a364e5a4f9644a08bdc8621c11f406c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/2a364e5a4f9644a08bdc8621c11f406c 2023-04-13 02:55:00,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1876aa97d55a4688954c9a586603397d 2023-04-13 02:55:00,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c54583ca627c4e62a50bf0e2545136a2 2023-04-13 02:55:00,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/7f713fb103a44dd194f4a92081c663b5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/7f713fb103a44dd194f4a92081c663b5 2023-04-13 02:55:00,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/dc8787be91f6459e90bc9af9e4a0ae41 2023-04-13 02:55:00,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6efc2b9393e347e9a091cbcb97a88a3e 2023-04-13 02:55:00,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/1bbf03490dad4bd1a3d7111265097fb2 2023-04-13 02:55:00,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/a4327a8954c340cab7b62972a8a4f610 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/a4327a8954c340cab7b62972a8a4f610 2023-04-13 02:55:00,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/57c46351c02642ff9bb0adad759ae1ac 2023-04-13 02:55:00,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/5d15a1ea0b054876b8a09be856e5b5e4 2023-04-13 02:55:00,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/00f50428b67f41a1a99c1be5bbf46f10 2023-04-13 02:55:00,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c67bacc62fcd4ccaa8a18e9575626efd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c67bacc62fcd4ccaa8a18e9575626efd 2023-04-13 02:55:00,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/8969d65478bb47a59766a671d767f613 2023-04-13 02:55:00,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/443a38400ee74d28b1d9dea66b910d59 2023-04-13 02:55:00,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/65fa05bbdfbb4f2f873253a9111f6bd7 2023-04-13 02:55:00,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/0d4484ab7643436e8d84c780a9243e9b 2023-04-13 02:55:00,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e1c079b33bb74d25ac0c9c0276cd6024 2023-04-13 02:55:00,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/da7f2daff6a9489ab631ccd464bcbd56 2023-04-13 02:55:00,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/28b0499bd3f94b0ca9e99b51dc034435 2023-04-13 02:55:00,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4ed1f37109634ed0a7d14283fc0b85af 2023-04-13 02:55:00,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/6f5ff7a4a8844e078777c39a999280ef 2023-04-13 02:55:00,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/b9dc0d9f7b18469d82fe2b484a70996c 2023-04-13 02:55:00,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e5858e13ca04441c82daa7dc502d6a3c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/b1716759c9624b008fa8bba667dd3b63, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/6f501904ac8142968c1e9cf30b356b8f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ef5ba2385c284c8ab7c933d60b7aae13, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/95c16a18826e4ef1b8fbfb57719ccf1a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22] to archive 2023-04-13 02:55:00,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:00,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/45480208791b42a2a74acac7f497229b 2023-04-13 02:55:00,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/02d47cd8fb6840a1a9c1eb905879b03a 2023-04-13 02:55:00,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1ce31a1465b54dd6ae1a6448280371cd 2023-04-13 02:55:00,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e5858e13ca04441c82daa7dc502d6a3c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e5858e13ca04441c82daa7dc502d6a3c 2023-04-13 02:55:00,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/1c965f00271647b39a2cb8bd5836e9a0 2023-04-13 02:55:00,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/c5c144793df44fcabff01ab9f701e883 2023-04-13 02:55:00,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ff4c6070446b461683034d7941f2bc4c 2023-04-13 02:55:00,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/b1716759c9624b008fa8bba667dd3b63 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/b1716759c9624b008fa8bba667dd3b63 2023-04-13 02:55:00,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/78304e7d7170407f9912f745c2e555b4 2023-04-13 02:55:00,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8b3c6f112dab41648667e35ac3c96762 2023-04-13 02:55:00,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/6f501904ac8142968c1e9cf30b356b8f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/6f501904ac8142968c1e9cf30b356b8f 2023-04-13 02:55:00,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d0315cb67b4140f59c98c39e57c35aa8 2023-04-13 02:55:00,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ba1b97cc86f248b3a6be4bf7a3161508 2023-04-13 02:55:00,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/0bbd5b6d91f94fa78106eadc5ea1af5c 2023-04-13 02:55:00,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ef5ba2385c284c8ab7c933d60b7aae13 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/ef5ba2385c284c8ab7c933d60b7aae13 2023-04-13 02:55:00,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/5f70381611b74898acaaa378c7bf7287 2023-04-13 02:55:00,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/8bb9b03846a64b42b9274d9707382114 2023-04-13 02:55:00,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/d1d40d1d3b97427f93641b06afefa005 2023-04-13 02:55:00,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/95c16a18826e4ef1b8fbfb57719ccf1a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/95c16a18826e4ef1b8fbfb57719ccf1a 2023-04-13 02:55:00,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/72adb7854aab42e58ec06c7f62366ec0 2023-04-13 02:55:00,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/98a901fd5293440382b9cfa3350e700a 2023-04-13 02:55:00,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/540069a287cc4f2db22cd8821ffb8476 2023-04-13 02:55:00,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/667138aa700f4389b6e44cda011ccb58 2023-04-13 02:55:00,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9ae9941110504c0aad9882275e218796 2023-04-13 02:55:00,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/a97445e89d3349dcb2da4d5bbe2771be 2023-04-13 02:55:00,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/024fccfe148e4fa48d73a5aa513f6a6f 2023-04-13 02:55:00,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/48d5e18003934a1ab213d2636a3895e2 2023-04-13 02:55:00,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/538bbb756e2b43f4aca09cb488add66a 2023-04-13 02:55:00,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e8d6dedeeeec4b3c9df6f011759f5e22 2023-04-13 02:55:00,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/recovered.edits/1349.seqid, newMaxSeqId=1349, maxSeqId=1 2023-04-13 02:55:00,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c. 2023-04-13 02:55:00,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for aa0d51e3ad17c60b2f3a2f2437425a6c: 2023-04-13 02:55:00,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:55:00,292 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=aa0d51e3ad17c60b2f3a2f2437425a6c, regionState=CLOSED 2023-04-13 02:55:00,293 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354500292"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354500292"}]},"ts":"1681354500292"} 2023-04-13 02:55:00,294 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-13 02:55:00,294 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure aa0d51e3ad17c60b2f3a2f2437425a6c, server=jenkins-hbase4.apache.org,34919,1681354416534 in 2.4920 sec 2023-04-13 02:55:00,295 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-13 02:55:00,295 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=aa0d51e3ad17c60b2f3a2f2437425a6c, UNASSIGN in 2.4960 sec 2023-04-13 02:55:00,296 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354500296"}]},"ts":"1681354500296"} 2023-04-13 02:55:00,297 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:55:00,298 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:55:00,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5070 sec 2023-04-13 02:55:01,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 02:55:01,902 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-13 02:55:01,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:55:01,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,905 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 02:55:01,906 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,911 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:55:01,914 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/recovered.edits] 2023-04-13 02:55:01,916 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/485b19dfb733455f94fc682e3e23bf88 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/485b19dfb733455f94fc682e3e23bf88 2023-04-13 02:55:01,917 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/ae6ddd4aa17745c7bc5dd13f069fdd54 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/ae6ddd4aa17745c7bc5dd13f069fdd54 2023-04-13 02:55:01,919 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/b38219c204624221b6328c425ab683f5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/b38219c204624221b6328c425ab683f5 2023-04-13 02:55:01,920 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f8012badf6c84491ad9155b23f07b987 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/A/f8012badf6c84491ad9155b23f07b987 2023-04-13 02:55:01,922 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4c18ee6eaa224d369d0b7db3dcb4df92 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/4c18ee6eaa224d369d0b7db3dcb4df92 2023-04-13 02:55:01,923 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bb0f45e3d1124ece9b95fe34c0327232 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/bb0f45e3d1124ece9b95fe34c0327232 2023-04-13 02:55:01,924 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c2046e84ccb94919ae5d822105afb7f0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/c2046e84ccb94919ae5d822105afb7f0 2023-04-13 02:55:01,925 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e90dc946701c4a2b81fa6590e42bd28f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/B/e90dc946701c4a2b81fa6590e42bd28f 2023-04-13 02:55:01,927 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/00b81cc287df4bf89376a172604ad42a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/00b81cc287df4bf89376a172604ad42a 2023-04-13 02:55:01,928 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/844afd0a0c3e4e3bbac67381f35a6c43 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/844afd0a0c3e4e3bbac67381f35a6c43 2023-04-13 02:55:01,929 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9905151273a64969a671e79101fc5911 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/9905151273a64969a671e79101fc5911 2023-04-13 02:55:01,930 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e91325764a604909976727e60419adba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/C/e91325764a604909976727e60419adba 2023-04-13 02:55:01,932 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/recovered.edits/1349.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c/recovered.edits/1349.seqid 2023-04-13 02:55:01,933 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/aa0d51e3ad17c60b2f3a2f2437425a6c 2023-04-13 02:55:01,933 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:01,935 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,936 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:55:01,938 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:55:01,939 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,939 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:55:01,940 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354501939"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:01,941 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:55:01,941 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => aa0d51e3ad17c60b2f3a2f2437425a6c, NAME => 'TestAcidGuarantees,,1681354476428.aa0d51e3ad17c60b2f3a2f2437425a6c.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:55:01,941 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:55:01,941 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354501941"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:01,943 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:55:01,945 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:01,946 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 42 msec 2023-04-13 02:55:02,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 02:55:02,007 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-13 02:55:02,018 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227 (was 231), OpenFileDescriptor=532 (was 534), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=351 (was 375), ProcessCount=169 (was 169), AvailableMemoryMB=850 (was 886) 2023-04-13 02:55:02,030 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227, OpenFileDescriptor=532, MaxFileDescriptor=60000, SystemLoadAverage=351, ProcessCount=169, AvailableMemoryMB=850 2023-04-13 02:55:02,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:55:02,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:55:02,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:02,035 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:55:02,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-13 02:55:02,035 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:55:02,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 02:55:02,037 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,037 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a empty. 2023-04-13 02:55:02,038 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,038 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:02,044 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:55:02,046 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6ffe91e2c6ce88591f4f17c4b5e2f40a, NAME => 'TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:55:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 02:55:02,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 02:55:02,452 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:02,452 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6ffe91e2c6ce88591f4f17c4b5e2f40a, disabling compactions & flushes 2023-04-13 02:55:02,452 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,452 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,452 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. after waiting 0 ms 2023-04-13 02:55:02,452 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,453 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:02,455 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:55:02,455 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354502455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354502455"}]},"ts":"1681354502455"} 2023-04-13 02:55:02,456 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:55:02,457 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:55:02,457 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354502457"}]},"ts":"1681354502457"} 2023-04-13 02:55:02,458 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:55:02,462 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, ASSIGN}] 2023-04-13 02:55:02,463 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, ASSIGN 2023-04-13 02:55:02,464 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:55:02,615 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:02,615 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354502615"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354502615"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354502615"}]},"ts":"1681354502615"} 2023-04-13 02:55:02,616 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:02,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 02:55:02,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6ffe91e2c6ce88591f4f17c4b5e2f40a, NAME => 'TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:55:02,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:02,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,773 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,774 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A 2023-04-13 02:55:02,774 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A 2023-04-13 02:55:02,774 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:02,775 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName A 2023-04-13 02:55:02,775 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:02,775 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,776 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B 2023-04-13 02:55:02,776 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B 2023-04-13 02:55:02,776 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:02,777 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName B 2023-04-13 02:55:02,777 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:02,777 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,778 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C 2023-04-13 02:55:02,778 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C 2023-04-13 02:55:02,778 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:02,778 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName C 2023-04-13 02:55:02,779 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:02,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:55:02,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:02,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:55:02,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6ffe91e2c6ce88591f4f17c4b5e2f40a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60052599, jitterRate=-0.10514654219150543}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:55:02,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:02,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., pid=37, masterSystemTime=1681354502768 2023-04-13 02:55:02,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:02,786 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:02,787 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354502786"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354502786"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354502786"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354502786"}]},"ts":"1681354502786"} 2023-04-13 02:55:02,789 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-13 02:55:02,789 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 in 171 msec 2023-04-13 02:55:02,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-13 02:55:02,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, ASSIGN in 327 msec 2023-04-13 02:55:02,790 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:55:02,791 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354502791"}]},"ts":"1681354502791"} 2023-04-13 02:55:02,791 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:55:02,793 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:55:02,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 761 msec 2023-04-13 02:55:03,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 02:55:03,140 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-13 02:55:03,140 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1cee66b5 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:03,145 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d800352, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:03,146 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:03,149 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48848, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:03,150 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:55:03,151 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:55:03,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:55:03,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:55:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:03,563 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:55:03,564 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:55:03,564 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:55:03,566 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 02:55:03,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, REOPEN/MOVE}] 2023-04-13 02:55:03,569 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, REOPEN/MOVE 2023-04-13 02:55:03,569 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:03,569 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354503569"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354503569"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354503569"}]},"ts":"1681354503569"} 2023-04-13 02:55:03,570 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:03,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:03,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6ffe91e2c6ce88591f4f17c4b5e2f40a, disabling compactions & flushes 2023-04-13 02:55:03,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:03,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:03,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. after waiting 0 ms 2023-04-13 02:55:03,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:03,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 02:55:03,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:03,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:03,727 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 6ffe91e2c6ce88591f4f17c4b5e2f40a to self. 2023-04-13 02:55:03,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:03,728 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=CLOSED 2023-04-13 02:55:03,729 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354503728"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354503728"}]},"ts":"1681354503728"} 2023-04-13 02:55:03,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-13 02:55:03,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 in 159 msec 2023-04-13 02:55:03,731 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=true 2023-04-13 02:55:03,881 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:03,881 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354503881"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354503881"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354503881"}]},"ts":"1681354503881"} 2023-04-13 02:55:03,883 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:04,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6ffe91e2c6ce88591f4f17c4b5e2f40a, NAME => 'TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:55:04,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:04,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,039 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,040 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A 2023-04-13 02:55:04,040 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A 2023-04-13 02:55:04,040 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:04,040 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName A 2023-04-13 02:55:04,041 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:04,042 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,042 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B 2023-04-13 02:55:04,042 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B 2023-04-13 02:55:04,043 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:04,043 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName B 2023-04-13 02:55:04,043 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:04,043 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,044 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C 2023-04-13 02:55:04,044 DEBUG [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C 2023-04-13 02:55:04,044 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:04,044 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ffe91e2c6ce88591f4f17c4b5e2f40a columnFamilyName C 2023-04-13 02:55:04,045 INFO [StoreOpener-6ffe91e2c6ce88591f4f17c4b5e2f40a-1] regionserver.HStore(310): Store=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:04,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:55:04,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6ffe91e2c6ce88591f4f17c4b5e2f40a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63046162, jitterRate=-0.06053897738456726}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:55:04,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:04,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., pid=42, masterSystemTime=1681354504034 2023-04-13 02:55:04,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,051 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,051 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354504051"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354504051"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354504051"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354504051"}]},"ts":"1681354504051"} 2023-04-13 02:55:04,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-13 02:55:04,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 in 169 msec 2023-04-13 02:55:04,054 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-13 02:55:04,054 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, REOPEN/MOVE in 485 msec 2023-04-13 02:55:04,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-13 02:55:04,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-13 02:55:04,057 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-04-13 02:55:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-13 02:55:04,058 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x4c1d19b3 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,063 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2329054b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,063 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x14e151fb to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,069 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30951255, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,069 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x455a88b1 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,074 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5baa6768, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,074 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x722b2407 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,078 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63a535f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,078 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x066bf91c to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,082 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d4bfeca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,082 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x313c7ad6 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,086 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43736f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,087 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x20d28b68 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,090 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72eaaf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,091 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x750d2a7d to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,097 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4130667f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,097 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x27313b38 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,101 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b73a361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,101 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x2ee4c750 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:04,106 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a2d14f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:04,111 DEBUG [hconnection-0x3edf217a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,113 DEBUG [hconnection-0x1c458620-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,113 DEBUG [hconnection-0xc3bf1b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,114 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,114 DEBUG [hconnection-0x4344d619-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,114 DEBUG [hconnection-0x560cbf86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,115 DEBUG [hconnection-0x48163b91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,115 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,115 DEBUG [hconnection-0x75035bad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,116 DEBUG [hconnection-0x5b2aa711-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,117 DEBUG [hconnection-0x7f859a6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,118 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,119 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,119 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,119 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,120 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,120 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,122 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,126 DEBUG [hconnection-0x4a2b6818-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:04,127 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:04,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,127 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,131 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,132 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:04,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,140 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:04,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:04,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:04,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:04,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:04,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,144 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,144 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,144 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,144 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,144 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,146 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,147 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc5defe[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,156 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,159 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,159 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,159 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,159 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,159 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,161 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,161 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,161 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354564158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,167 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,167 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354564161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354564163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354564164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354564165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,267 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354564266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354564269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,272 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,273 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,274 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354564270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354564270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354564271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,287 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd7e40d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,290 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,290 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,299 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,300 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,300 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,300 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,303 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,306 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,307 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,307 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,407 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,410 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,412 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,414 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,414 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,414 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,414 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,420 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7992e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,425 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,428 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,428 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,438 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,438 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354564469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354564471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354564476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354564477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354564478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,538 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,543 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1cb13d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,551 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,553 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,556 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,556 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,558 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041390bb87840b104daeb778666304ce4f97 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041390bb87840b104daeb778666304ce4f97 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,563 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,563 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/5851df8f9eed482a92234bb6cb893020 2023-04-13 02:55:04,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,563 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,564 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,564 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,664 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,668 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,671 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,671 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,671 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,671 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,674 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2fac8e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,677 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,681 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,692 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,692 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354564772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354564776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354564780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354564781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:04,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354564782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,792 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ad64a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,809 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,809 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,810 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,810 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,810 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,816 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,816 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,916 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:04,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:04,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:04,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:04,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:04,919 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:04,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:04,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:04,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:04,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:04,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:04,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:04,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:04,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:04,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:04,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:04,924 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:04,924 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:04,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:04,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:04,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:04,924 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:04,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:04,927 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:04,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34084680[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:04,927 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:04,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:04,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:04,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:04,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:04,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:04,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:04,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:04,933 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:04,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:04,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:04,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:04,933 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:04,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:04,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:04,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:04,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:04,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:04,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:04,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:04,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:04,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:04,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:04,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:04,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:04,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:04,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:04,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:04,940 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:04,940 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:04,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:04,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/f2e5806321264baa9cfa525e3d3d4ff7 2023-04-13 02:55:04,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/69eaf52f8ba548b187770f9f40a9fbb7 2023-04-13 02:55:04,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/5851df8f9eed482a92234bb6cb893020 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020 2023-04-13 02:55:05,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020, entries=200, sequenceid=16, filesize=31.9 K 2023-04-13 02:55:05,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/f2e5806321264baa9cfa525e3d3d4ff7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7 2023-04-13 02:55:05,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7, entries=150, sequenceid=16, filesize=11.6 K 2023-04-13 02:55:05,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/69eaf52f8ba548b187770f9f40a9fbb7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7 2023-04-13 02:55:05,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7, entries=150, sequenceid=16, filesize=11.6 K 2023-04-13 02:55:05,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 875ms, sequenceid=16, compaction requested=false 2023-04-13 02:55:05,012 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 02:55:05,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:05,040 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:05,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:05,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:05,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:05,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:05,043 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:05,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:05,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:05,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:05,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:05,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:05,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:05,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:05,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:05,047 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:05,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:05,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354565289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354565290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354565290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354565291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354565292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354565393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354565398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354565398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354565398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354565398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,460 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ef737cd84a314ef2be57acf7bc675978 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ef737cd84a314ef2be57acf7bc675978 2023-04-13 02:55:05,465 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/4fc3abaac5234c2ab6422a693921d919 2023-04-13 02:55:05,500 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/303b3568f2354255a7c967eb77d5c886 2023-04-13 02:55:05,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354565597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354565602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354565604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354565604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354565605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,823 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:55:05,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354565900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354565907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,912 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/10afe8fa47ef4bfcb0a3623c89a39e54 2023-04-13 02:55:05,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354565908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354565910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:05,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354565910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,916 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/4fc3abaac5234c2ab6422a693921d919 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919 2023-04-13 02:55:05,919 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919, entries=150, sequenceid=36, filesize=25.1 K 2023-04-13 02:55:05,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/303b3568f2354255a7c967eb77d5c886 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886 2023-04-13 02:55:05,925 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 02:55:05,925 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/10afe8fa47ef4bfcb0a3623c89a39e54 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54 2023-04-13 02:55:05,929 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 02:55:05,929 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 882ms, sequenceid=36, compaction requested=false 2023-04-13 02:55:05,930 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:05,930 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:05,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:05,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:05,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:05,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:05,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:05,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a739c65[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:05,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:05,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:05,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:05,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:05,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:05,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:05,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:05,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:05,943 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:05,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:05,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:05,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:05,943 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:05,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:05,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:05,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:05,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:05,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:05,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:05,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:05,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:05,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:05,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:05,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:05,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:05,953 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:05,953 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,953 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:05,953 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:05,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:05,953 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:05,953 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:05,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:06,053 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:06,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:06,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:06,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:06,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:06,056 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:06,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:06,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:06,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:06,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:06,057 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:06,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:06,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:06,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:06,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:06,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:06,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:06,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:06,060 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:06,060 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:06,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:06,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:06,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:06,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354566432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354566432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354566433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354566439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354566440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,476 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e534d836c5ca4cfe8678e6c0de6893f4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e534d836c5ca4cfe8678e6c0de6893f4 2023-04-13 02:55:06,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354566541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354566542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354566542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354566548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354566549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354566745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354566745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354566749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354566754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:06,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354566755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:06,881 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/2a5b87b54d4240178972a77da12209eb 2023-04-13 02:55:07,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354567052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354567053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354567053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354567058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354567060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,298 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/48ca9f1a8589473cb53efa8de84970b7 2023-04-13 02:55:07,308 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/a92afd18d4754cb18bae5c8415c56015 2023-04-13 02:55:07,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/2a5b87b54d4240178972a77da12209eb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb 2023-04-13 02:55:07,319 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb, entries=150, sequenceid=48, filesize=25.1 K 2023-04-13 02:55:07,320 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/48ca9f1a8589473cb53efa8de84970b7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7 2023-04-13 02:55:07,323 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7, entries=150, sequenceid=48, filesize=11.6 K 2023-04-13 02:55:07,325 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/a92afd18d4754cb18bae5c8415c56015 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015 2023-04-13 02:55:07,333 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015, entries=150, sequenceid=48, filesize=11.6 K 2023-04-13 02:55:07,334 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1274ms, sequenceid=48, compaction requested=true 2023-04-13 02:55:07,334 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:07,334 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:07,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:07,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:07,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:07,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,335 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:07,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4f0079[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:07,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:07,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:07,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:07,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:07,344 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:07,344 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,344 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:07,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:07,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:07,347 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:07,347 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:07,347 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:07,347 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:07,347 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:07,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,349 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,350 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:07,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,352 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,352 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:07,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,452 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:07,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:07,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:07,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:07,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:07,456 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:07,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:07,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:07,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,458 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:07,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:07,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:07,459 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:07,459 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:07,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:07,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:07,460 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:07,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,478 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413286747edcf0e4e3ea291405afac62ab5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413286747edcf0e4e3ea291405afac62ab5 2023-04-13 02:55:07,484 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a0cc7c371c3643f9a05f9134cbc196f2 2023-04-13 02:55:07,494 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/462774ca7e2a417ab8fbb1e37bebc5cb 2023-04-13 02:55:07,505 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/9fef76d50aec4b7dbfc58f7a91c9ad1c 2023-04-13 02:55:07,509 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a0cc7c371c3643f9a05f9134cbc196f2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2 2023-04-13 02:55:07,513 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2, entries=150, sequenceid=68, filesize=25.1 K 2023-04-13 02:55:07,514 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/462774ca7e2a417ab8fbb1e37bebc5cb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb 2023-04-13 02:55:07,517 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb, entries=150, sequenceid=68, filesize=11.6 K 2023-04-13 02:55:07,518 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/9fef76d50aec4b7dbfc58f7a91c9ad1c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c 2023-04-13 02:55:07,521 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c, entries=150, sequenceid=68, filesize=11.6 K 2023-04-13 02:55:07,523 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 63ms, sequenceid=68, compaction requested=true 2023-04-13 02:55:07,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:07,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:07,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:07,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43795b1c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:07,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:07,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:07,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,531 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:07,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:07,535 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:07,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:07,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:07,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:07,535 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:07,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:07,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:07,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,556 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,557 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:07,557 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:07,657 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:07,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:07,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:07,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:07,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:07,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:07,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:07,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:07,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:07,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:07,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:07,661 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:07,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:07,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:07,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:07,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:07,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:07,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:07,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:07,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:07,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:07,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:07,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:07,669 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,669 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,669 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:07,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b339b5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:07,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:07,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:07,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:07,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:07,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:07,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:07,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:07,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:07,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:07,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:07,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,680 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:07,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:07,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:07,686 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:07,686 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:07,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354567698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354567697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354567699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354507688, max timestamp=1681354507704Num uniques 859; 2023-04-13 02:55:07,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:07,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354507688, max timestamp=1681354507704Num uniques 859; 2023-04-13 02:55:07,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681354507688, max timestamp=1681354507704Num uniques 859; 2023-04-13 02:55:07,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:07,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:07,786 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:07,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:07,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354567777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:07,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:07,790 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:07,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:07,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:07,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:07,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354567786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:07,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:07,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:07,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:07,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:07,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:07,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:07,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:07,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:07,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:07,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:07,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:07,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:07,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e51bd2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:07,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:07,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,802 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:07,804 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:07,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:07,806 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:07,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:07,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:07,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:07,806 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:07,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:07,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,812 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:07,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,812 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:07,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354567817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354567817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354567817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354567890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:07,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354567896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,913 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:07,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:07,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:07,916 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:07,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:07,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:07,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:07,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:07,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:07,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:07,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:07,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,921 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:07,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fc811f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:07,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:07,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:07,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:07,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:07,927 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:07,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:07,929 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:07,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:07,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:07,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:07,930 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:07,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:07,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:07,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:07,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:07,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:07,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:07,935 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:07,936 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:07,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:07,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:07,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:07,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:08,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354568021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354568021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354568022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,036 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,039 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,041 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,041 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,043 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,045 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f548f67[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,052 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,052 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,058 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,059 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,059 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,075 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041324006cc9693e4532b59e72e91f158f57 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324006cc9693e4532b59e72e91f158f57 2023-04-13 02:55:08,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354568097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354568101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,159 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,162 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,167 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,168 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,168 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,169 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,169 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,169 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,169 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,169 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,169 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,171 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,172 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38675a9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,178 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,178 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,187 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,187 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,187 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,287 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,290 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,294 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,295 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5866c580[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,299 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,302 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,304 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,304 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,306 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,306 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,310 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,310 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,310 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,310 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354568325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354568325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354568330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354568406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354568406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,411 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,414 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,420 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,420 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,421 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad053ea[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,424 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,424 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,428 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,428 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,430 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:08,436 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,436 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,480 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=79, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/7162ec7bcd9f47b8b44eb8c35f191664 2023-04-13 02:55:08,537 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f4d860[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,552 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,556 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,556 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,562 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,562 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,563 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,563 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,663 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,666 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,667 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21efdcca[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,681 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,681 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,688 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,689 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:08,789 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,792 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,796 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4b47b2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,802 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,806 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,812 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:08,812 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:08,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354568831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354568834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354568840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/98ad2b297f794837950754b2d5cbe642 2023-04-13 02:55:08,913 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:08,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:08,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:08,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:08,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:08,915 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:08,915 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:08,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354568912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:08,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:08,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:08,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:08,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:08,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:08,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354568913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:08,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@571c02b2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:08,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:08,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,924 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:08,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:08,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:08,934 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:08,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:08,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:08,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:08,934 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:08,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:08,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:08,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:08,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:08,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:08,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,940 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:08,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:08,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:08,940 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:09,041 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:09,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:09,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:09,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:09,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:09,044 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:09,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:09,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:09,048 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:09,048 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,048 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:09,048 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:09,049 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:09,051 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:09,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601ce7e9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:09,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:09,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:09,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:09,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:09,055 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:09,055 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,055 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,055 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:09,058 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:09,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:09,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:09,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:09,058 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:09,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,061 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:09,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:09,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,065 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:09,065 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:09,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,165 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:09,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:09,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:09,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:09,169 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:09,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:09,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:09,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:09,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:09,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:09,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:09,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:09,173 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:09,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:09,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,175 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:09,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:09,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fca6eac[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:09,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,178 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:09,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:09,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:09,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:09,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:09,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,181 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:09,183 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:09,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:09,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:09,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:09,183 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:09,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,184 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:09,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:09,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,189 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:09,189 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:09,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:09,289 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:09,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:09,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:09,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:09,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:09,293 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:09,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:09,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,296 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:09,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:09,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:09,297 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:09,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:09,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:09,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/65de4f418e634fde863a1684d8821642 2023-04-13 02:55:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:09,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa9d830[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:09,302 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:09,302 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,307 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:09,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:09,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:09,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:09,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:09,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:09,311 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:09,311 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:09,311 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:09,311 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:09,311 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:09,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/7162ec7bcd9f47b8b44eb8c35f191664 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664 2023-04-13 02:55:09,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:09,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:09,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:09,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664, entries=400, sequenceid=79, filesize=59.0 K 2023-04-13 02:55:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:09,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/98ad2b297f794837950754b2d5cbe642 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642 2023-04-13 02:55:09,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642, entries=150, sequenceid=79, filesize=11.6 K 2023-04-13 02:55:09,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:09,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:09,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:09,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/65de4f418e634fde863a1684d8821642 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642 2023-04-13 02:55:09,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642, entries=150, sequenceid=79, filesize=11.6 K 2023-04-13 02:55:09,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1671ms, sequenceid=79, compaction requested=true 2023-04-13 02:55:09,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:09,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:09,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:09,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:09,328 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:09,329 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:09,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:09,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:09,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,329 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:09,329 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:09,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:09,330 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 170378 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:09,330 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:09,330 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/A is initiating minor compaction (all files) 2023-04-13 02:55:09,330 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/B is initiating minor compaction (all files) 2023-04-13 02:55:09,330 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/A in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,330 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/B in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,330 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=166.4 K 2023-04-13 02:55:09,330 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=58.0 K 2023-04-13 02:55:09,331 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5851df8f9eed482a92234bb6cb893020, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354504130 2023-04-13 02:55:09,331 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f2e5806321264baa9cfa525e3d3d4ff7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354504130 2023-04-13 02:55:09,331 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4fc3abaac5234c2ab6422a693921d919, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681354504158 2023-04-13 02:55:09,331 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 303b3568f2354255a7c967eb77d5c886, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681354504158 2023-04-13 02:55:09,331 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2a5b87b54d4240178972a77da12209eb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354505289 2023-04-13 02:55:09,332 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 48ca9f1a8589473cb53efa8de84970b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354505289 2023-04-13 02:55:09,332 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a0cc7c371c3643f9a05f9134cbc196f2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681354506430 2023-04-13 02:55:09,332 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 462774ca7e2a417ab8fbb1e37bebc5cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681354506430 2023-04-13 02:55:09,332 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 98ad2b297f794837950754b2d5cbe642, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507644 2023-04-13 02:55:09,332 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7162ec7bcd9f47b8b44eb8c35f191664, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507641 2023-04-13 02:55:09,341 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:09,345 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#A#compaction#295 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:09,429 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:09,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:09,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:09,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:09,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:09,434 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:09,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:09,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:09,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,436 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:09,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:09,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:09,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:09,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:09,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:09,438 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,438 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:09,438 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:55:09,438 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:09,438 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:09,439 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:09,439 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:09,439 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:09,439 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:09,751 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/93862df34c7c4bd38c6ecebaaa296a75 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/93862df34c7c4bd38c6ecebaaa296a75 2023-04-13 02:55:09,755 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/B of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 93862df34c7c4bd38c6ecebaaa296a75(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:09,755 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:09,756 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, priority=11, startTime=1681354509328; duration=0sec 2023-04-13 02:55:09,756 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:09,756 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:09,757 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:09,757 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/C is initiating minor compaction (all files) 2023-04-13 02:55:09,757 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/C in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:09,757 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=58.0 K 2023-04-13 02:55:09,758 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 69eaf52f8ba548b187770f9f40a9fbb7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354504130 2023-04-13 02:55:09,758 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 10afe8fa47ef4bfcb0a3623c89a39e54, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681354504158 2023-04-13 02:55:09,758 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a92afd18d4754cb18bae5c8415c56015, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354505289 2023-04-13 02:55:09,759 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9fef76d50aec4b7dbfc58f7a91c9ad1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681354506430 2023-04-13 02:55:09,759 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 65de4f418e634fde863a1684d8821642, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507644 2023-04-13 02:55:09,765 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/fb1402df26af49228459c4e88fa10862 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fb1402df26af49228459c4e88fa10862 2023-04-13 02:55:09,769 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#C#compaction#297 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:09,771 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/A of 6ffe91e2c6ce88591f4f17c4b5e2f40a into fb1402df26af49228459c4e88fa10862(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:09,771 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:09,771 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, priority=11, startTime=1681354509328; duration=0sec 2023-04-13 02:55:09,771 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:09,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:09,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:09,854 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b9222bb1931d43f58c2da64b8808c1df to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b9222bb1931d43f58c2da64b8808c1df 2023-04-13 02:55:09,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354569863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354569864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354569866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354569917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354569929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354569970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354569970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:09,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:09,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354569970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354570175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354570175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354570175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,184 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/a9c67f8e26064eceb2c3394d954d2915 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a9c67f8e26064eceb2c3394d954d2915 2023-04-13 02:55:10,189 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/C of 6ffe91e2c6ce88591f4f17c4b5e2f40a into a9c67f8e26064eceb2c3394d954d2915(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:10,189 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:10,189 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, priority=11, startTime=1681354509328; duration=0sec 2023-04-13 02:55:10,189 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:10,260 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/cab9cec650c04b3089bcb914f7c24028 2023-04-13 02:55:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354570482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354570482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354570482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,670 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/de8c4f2f056d462288bcca720a1b9808 2023-04-13 02:55:10,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354570986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354570986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:10,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:10,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354570988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,081 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/c91c5f686dbf4aaca429153446b1ead0 2023-04-13 02:55:11,086 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/cab9cec650c04b3089bcb914f7c24028 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028 2023-04-13 02:55:11,090 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028, entries=150, sequenceid=114, filesize=25.1 K 2023-04-13 02:55:11,091 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/de8c4f2f056d462288bcca720a1b9808 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808 2023-04-13 02:55:11,094 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808, entries=150, sequenceid=114, filesize=11.6 K 2023-04-13 02:55:11,095 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/c91c5f686dbf4aaca429153446b1ead0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0 2023-04-13 02:55:11,100 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0, entries=150, sequenceid=114, filesize=11.6 K 2023-04-13 02:55:11,100 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1662ms, sequenceid=114, compaction requested=false 2023-04-13 02:55:11,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:11,101 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:11,101 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:11,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:11,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:11,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:11,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d175645[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:11,104 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:11,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,106 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:11,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:11,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:11,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:11,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:11,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:11,110 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:11,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:11,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:11,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:11,110 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:11,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:11,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:11,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:11,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,120 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:11,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:11,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:11,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:11,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:11,123 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:11,123 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:11,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:11,223 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:11,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:11,226 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:11,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:11,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:11,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:11,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:11,227 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,228 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:11,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:11,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:11,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:11,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:11,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:11,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:11,229 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:11,229 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:11,229 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:11,229 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:11,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:11,646 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413059172fb37b845cea59146408452f235 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413059172fb37b845cea59146408452f235 2023-04-13 02:55:11,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:11,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:11,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354511952, max timestamp=1681354511976Num uniques 859; 2023-04-13 02:55:11,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:11,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354511952, max timestamp=1681354511976Num uniques 859; 2023-04-13 02:55:11,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354511952, max timestamp=1681354511976Num uniques 859; 2023-04-13 02:55:11,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:11,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:12,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354572004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354572009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354572010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354572010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354572012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,053 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/546e279f90b1417ab89d18fa7163aaef 2023-04-13 02:55:12,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354572112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354572120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354572122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354572122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354572122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354572319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354572324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354572327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354572327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354572328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,463 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/d4fb4dc01444489ab6af742b7b1a862b 2023-04-13 02:55:12,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354572625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354572629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354572634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354572634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354572635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,872 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/a5a6c9a6a4154e2ca9f16196f48a5399 2023-04-13 02:55:12,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/546e279f90b1417ab89d18fa7163aaef as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef 2023-04-13 02:55:12,879 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef, entries=150, sequenceid=128, filesize=25.1 K 2023-04-13 02:55:12,880 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/d4fb4dc01444489ab6af742b7b1a862b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b 2023-04-13 02:55:12,883 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b, entries=150, sequenceid=128, filesize=11.6 K 2023-04-13 02:55:12,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/a5a6c9a6a4154e2ca9f16196f48a5399 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399 2023-04-13 02:55:12,887 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399, entries=150, sequenceid=128, filesize=11.6 K 2023-04-13 02:55:12,888 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1659ms, sequenceid=128, compaction requested=true 2023-04-13 02:55:12,888 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:12,888 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:12,888 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:12,888 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:12,888 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:12,889 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,890 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab3d9a1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:12,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:12,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,894 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:12,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:12,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:12,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:12,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:12,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:12,898 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:12,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:12,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:12,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:12,898 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:12,899 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,899 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:12,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,899 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:12,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:12,900 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,901 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:12,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:12,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:12,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:12,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:12,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:12,903 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:12,903 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:12,903 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:12,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:13,004 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:13,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:13,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:13,007 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:13,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:13,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:13,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:13,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,007 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,009 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:13,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:13,010 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,010 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:13,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:13,011 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:13,011 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:13,011 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:13,011 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:13,012 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:13,026 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f8658a25fba54863a855a563ee53eff5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f8658a25fba54863a855a563ee53eff5 2023-04-13 02:55:13,040 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/fa0ad743565c4260a327e46d55a67849 2023-04-13 02:55:13,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:13,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:13,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354573143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354573143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354573144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354573145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354573147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354573248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354573248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354573251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354573251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354573253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,454 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/38f20c9af1844cea9b804d330097f5e0 2023-04-13 02:55:13,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354573453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354573453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354573457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354573458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354573458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,464 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/f6cb313a488f41bea4c8e30d0c026470 2023-04-13 02:55:13,469 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/fa0ad743565c4260a327e46d55a67849 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849 2023-04-13 02:55:13,473 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849, entries=150, sequenceid=163, filesize=25.3 K 2023-04-13 02:55:13,474 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/38f20c9af1844cea9b804d330097f5e0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0 2023-04-13 02:55:13,477 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0, entries=150, sequenceid=163, filesize=11.8 K 2023-04-13 02:55:13,478 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/f6cb313a488f41bea4c8e30d0c026470 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470 2023-04-13 02:55:13,487 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470, entries=150, sequenceid=163, filesize=11.8 K 2023-04-13 02:55:13,488 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 477ms, sequenceid=163, compaction requested=true 2023-04-13 02:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:13,488 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:13,490 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:13,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:13,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e07b55[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:13,493 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:13,493 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,494 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:13,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:13,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:13,495 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:13,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:13,499 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:13,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:13,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:13,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:13,499 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:13,502 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,502 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:13,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,502 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:13,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:13,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:13,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:13,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:13,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:13,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:13,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:13,510 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:13,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:13,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:13,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,510 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:13,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:13,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:13,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:13,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,611 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:13,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:13,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:13,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:13,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:13,615 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:13,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:13,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:13,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:13,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:13,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:13,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,617 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:13,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:13,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:13,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:13,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:13,620 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:13,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:13,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:13,620 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:55:13,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:13,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:13,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:13,621 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:13,621 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:13,621 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:13,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:13,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:13,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354573851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354573851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354573852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354513830, max timestamp=1681354513862Num uniques 859; 2023-04-13 02:55:13,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354513830, max timestamp=1681354513862Num uniques 859; 2023-04-13 02:55:13,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354573862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:13,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354513830, max timestamp=1681354513862Num uniques 859; 2023-04-13 02:55:13,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:13,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:13,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354573965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354573965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354573965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354573966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:13,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:13,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354573971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,050 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304139c8bc1e0a1584d8a9afcec64daccb319 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139c8bc1e0a1584d8a9afcec64daccb319 2023-04-13 02:55:14,057 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/b99ab9c70b9743f88f98802122dae80a 2023-04-13 02:55:14,069 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/249d62fe54be4358bf08817dff428afe 2023-04-13 02:55:14,080 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/7653d945faa249cba3f4f5f0d296da51 2023-04-13 02:55:14,088 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/b99ab9c70b9743f88f98802122dae80a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a 2023-04-13 02:55:14,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354574083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,093 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a, entries=100, sequenceid=174, filesize=18.5 K 2023-04-13 02:55:14,095 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/249d62fe54be4358bf08817dff428afe as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe 2023-04-13 02:55:14,098 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe, entries=100, sequenceid=174, filesize=9.4 K 2023-04-13 02:55:14,100 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/7653d945faa249cba3f4f5f0d296da51 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51 2023-04-13 02:55:14,104 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51, entries=100, sequenceid=174, filesize=9.4 K 2023-04-13 02:55:14,105 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 485ms, sequenceid=174, compaction requested=true 2023-04-13 02:55:14,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:14,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,108 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@126c45e1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,110 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,118 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,119 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,120 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,125 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,125 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,126 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:14,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:14,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:14,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354574186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354574190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354574191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354574191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,226 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,229 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,231 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,235 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356e8c97[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,240 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,242 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,243 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,248 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,248 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,248 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:14,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354574290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354574292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354574296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354574296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354574301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,349 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,352 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,354 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,359 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,359 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,359 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,359 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3114254a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,362 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,362 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,362 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,363 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,366 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,366 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,368 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,369 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,373 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,373 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,374 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,474 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,477 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,478 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,478 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,481 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,481 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,482 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,482 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,483 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,483 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,485 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445bcaee[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,487 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,487 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,490 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,491 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,494 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,494 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,500 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,500 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,500 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,500 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,500 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:14,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354574497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354574504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354574505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354574509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,597 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041385d633e067f7453ebd1db715301e48dd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385d633e067f7453ebd1db715301e48dd 2023-04-13 02:55:14,600 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,604 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/6ca3f08c255e445c94432281abcefc86 2023-04-13 02:55:14,604 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354574597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,607 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,609 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,609 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,609 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,609 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,609 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,611 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,611 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,611 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e8ccc9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,614 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,614 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,615 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,615 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,616 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,616 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,619 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,619 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,619 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,619 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,619 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,625 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,625 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,626 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,626 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,726 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,730 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,730 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,730 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,730 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,730 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,732 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,732 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,734 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,734 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,734 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,734 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,736 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,736 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,737 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,737 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3e4f7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,739 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,741 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,741 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,741 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,742 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,742 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,742 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,743 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,748 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,748 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,748 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,748 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,748 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,749 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,749 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,749 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,750 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,753 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,754 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,754 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,754 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354574804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354574812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354574814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354574815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,854 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,857 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,860 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,863 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,863 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,863 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,863 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,863 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,867 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,868 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,868 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650db17c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,870 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,871 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,873 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,873 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,873 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,873 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:14,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:14,875 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:14,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:14,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:14,875 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:14,875 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:14,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,876 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,878 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:14,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:14,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,882 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:14,882 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,882 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,882 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:14,882 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:14,982 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:14,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:14,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:14,985 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:14,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:14,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:14,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,987 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:14,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:14,988 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,988 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:14,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:14,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:14,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:14,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:14,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:14,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:14,993 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,993 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:14,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9b0b89[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:14,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:14,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:14,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:14,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:14,996 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:14,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:14,997 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:15,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:15,000 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:15,000 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:15,000 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:15,000 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:15,000 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:15,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,003 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,005 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,008 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:15,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:15,008 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:15,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,009 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,009 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:15,009 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:15,009 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,009 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,009 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/7836b53b5df14459937a2b11301f2451 2023-04-13 02:55:15,109 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:15,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:15,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:15,112 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:15,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:15,112 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:15,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:15,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41178 deadline: 1681354575109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:15,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:15,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:15,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:15,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:15,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:15,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:15,119 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:15,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@de71e37[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:15,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:15,121 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:15,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:15,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:15,128 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:15,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:15,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:15,128 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:15,129 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:15,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:15,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,134 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,134 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:15,135 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:15,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,235 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:15,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:15,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:15,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:15,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:15,238 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:15,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:15,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:15,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,240 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:15,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:15,242 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,242 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:15,242 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:15,242 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:15,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:15,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:15,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6789c720[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:15,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:15,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,247 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:15,247 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:15,247 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:15,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:15,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:15,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:15,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:15,255 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:15,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:15,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:15,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:15,256 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:15,257 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,257 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,257 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,258 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:15,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:15,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,261 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:15,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,261 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:15,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,262 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:15,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354575312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354575319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354575322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354575322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,362 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:15,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:15,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:15,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:15,365 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:15,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:15,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:15,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:15,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:15,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:15,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:15,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:15,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:15,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:15,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:15,369 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:15,369 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:15,371 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,371 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:15,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f070cfc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:15,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:15,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:15,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:15,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:15,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:15,376 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:15,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:15,378 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:15,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:15,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:15,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:15,378 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:15,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:15,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,381 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:15,381 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,385 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,385 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:15,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,385 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:15,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:15,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,386 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:15,386 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:15,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:15,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/f7666edcb93f41048c972b1ec20fcd28 2023-04-13 02:55:15,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/6ca3f08c255e445c94432281abcefc86 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86 2023-04-13 02:55:15,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86, entries=200, sequenceid=210, filesize=32.1 K 2023-04-13 02:55:15,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/7836b53b5df14459937a2b11301f2451 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451 2023-04-13 02:55:15,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451, entries=150, sequenceid=210, filesize=11.8 K 2023-04-13 02:55:15,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/f7666edcb93f41048c972b1ec20fcd28 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28 2023-04-13 02:55:15,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28, entries=150, sequenceid=210, filesize=11.8 K 2023-04-13 02:55:15,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1268ms, sequenceid=210, compaction requested=true 2023-04-13 02:55:15,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:15,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:15,447 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:55:15,447 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:55:15,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:15,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:15,450 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155090 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:55:15,450 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/A is initiating minor compaction (all files) 2023-04-13 02:55:15,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69544 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:55:15,450 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/A in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/B is initiating minor compaction (all files) 2023-04-13 02:55:15,450 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/B in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,450 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fb1402df26af49228459c4e88fa10862, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=151.5 K 2023-04-13 02:55:15,450 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/93862df34c7c4bd38c6ecebaaa296a75, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=67.9 K 2023-04-13 02:55:15,451 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting fb1402df26af49228459c4e88fa10862, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507644 2023-04-13 02:55:15,451 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 93862df34c7c4bd38c6ecebaaa296a75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507644 2023-04-13 02:55:15,451 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting cab9cec650c04b3089bcb914f7c24028, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681354507757 2023-04-13 02:55:15,451 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting de8c4f2f056d462288bcca720a1b9808, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681354507757 2023-04-13 02:55:15,452 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 546e279f90b1417ab89d18fa7163aaef, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681354509858 2023-04-13 02:55:15,452 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d4fb4dc01444489ab6af742b7b1a862b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681354509858 2023-04-13 02:55:15,452 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting fa0ad743565c4260a327e46d55a67849, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681354511976 2023-04-13 02:55:15,452 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 38f20c9af1844cea9b804d330097f5e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681354511976 2023-04-13 02:55:15,453 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b99ab9c70b9743f88f98802122dae80a, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681354513143 2023-04-13 02:55:15,453 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 249d62fe54be4358bf08817dff428afe, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681354513143 2023-04-13 02:55:15,453 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6ca3f08c255e445c94432281abcefc86, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:15,453 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7836b53b5df14459937a2b11301f2451, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:15,466 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#B#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:15,484 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#A#compaction#313 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:15,486 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:15,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:15,488 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/bf9fb67759e14579b98157f7dcbc523e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bf9fb67759e14579b98157f7dcbc523e 2023-04-13 02:55:15,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:15,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:15,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:15,494 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,494 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,495 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:15,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:15,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,496 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/B of 6ffe91e2c6ce88591f4f17c4b5e2f40a into bf9fb67759e14579b98157f7dcbc523e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:15,496 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:15,496 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, priority=10, startTime=1681354515447; duration=0sec 2023-04-13 02:55:15,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:15,496 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:15,496 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:15,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:15,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:15,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:15,498 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69544 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:55:15,498 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/C is initiating minor compaction (all files) 2023-04-13 02:55:15,498 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/C in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:15,498 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a9c67f8e26064eceb2c3394d954d2915, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=67.9 K 2023-04-13 02:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:15,498 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:15,499 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a9c67f8e26064eceb2c3394d954d2915, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681354507644 2023-04-13 02:55:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:15,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:15,499 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:15,499 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:15,499 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c91c5f686dbf4aaca429153446b1ead0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1681354507757 2023-04-13 02:55:15,499 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:15,499 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a5a6c9a6a4154e2ca9f16196f48a5399, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681354509858 2023-04-13 02:55:15,499 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:15,500 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:15,500 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:15,500 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:15,500 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f6cb313a488f41bea4c8e30d0c026470, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681354511976 2023-04-13 02:55:15,500 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:15,500 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:15,500 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7653d945faa249cba3f4f5f0d296da51, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681354513143 2023-04-13 02:55:15,500 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f7666edcb93f41048c972b1ec20fcd28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:15,513 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#C#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:15,527 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/cbf5ba8779ca4fcb9b768721aaf716b9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/cbf5ba8779ca4fcb9b768721aaf716b9 2023-04-13 02:55:15,531 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/C of 6ffe91e2c6ce88591f4f17c4b5e2f40a into cbf5ba8779ca4fcb9b768721aaf716b9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:15,531 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:15,531 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, priority=10, startTime=1681354515447; duration=0sec 2023-04-13 02:55:15,531 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:15,929 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413df1e0090cbdd4d74b2ddf9183b224a24 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df1e0090cbdd4d74b2ddf9183b224a24 2023-04-13 02:55:16,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:16,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:16,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354516169, max timestamp=1681354516207Num uniques 859; 2023-04-13 02:55:16,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354516169, max timestamp=1681354516207Num uniques 859; 2023-04-13 02:55:16,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:16,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354516169, max timestamp=1681354516207Num uniques 859; 2023-04-13 02:55:16,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:16,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:16,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354576319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354576321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,335 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/107f922390c04f918f3229a71e7b2c41 2023-04-13 02:55:16,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354576333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354576333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354576334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354576424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354576627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,721 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/b86eae5c4dd44b2aba26002d839fd401 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b86eae5c4dd44b2aba26002d839fd401 2023-04-13 02:55:16,725 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/A of 6ffe91e2c6ce88591f4f17c4b5e2f40a into b86eae5c4dd44b2aba26002d839fd401(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 02:55:16,725 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:16,725 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, priority=10, startTime=1681354515447; duration=1sec 2023-04-13 02:55:16,725 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:16,745 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/dd45578293a9484f8ce11487bebd4d79 2023-04-13 02:55:16,756 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/ec91e557ebe249638f98309e274749af 2023-04-13 02:55:16,760 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/107f922390c04f918f3229a71e7b2c41 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41 2023-04-13 02:55:16,768 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41, entries=150, sequenceid=221, filesize=25.3 K 2023-04-13 02:55:16,769 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/dd45578293a9484f8ce11487bebd4d79 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79 2023-04-13 02:55:16,773 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79, entries=150, sequenceid=221, filesize=11.8 K 2023-04-13 02:55:16,773 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/ec91e557ebe249638f98309e274749af as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af 2023-04-13 02:55:16,777 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af, entries=150, sequenceid=221, filesize=11.8 K 2023-04-13 02:55:16,778 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1279ms, sequenceid=221, compaction requested=false 2023-04-13 02:55:16,778 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:16,778 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:16,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:16,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:16,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:16,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:16,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:16,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:16,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e795d6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:16,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:16,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,785 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,785 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:16,785 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:16,785 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:16,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:16,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:16,787 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:16,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:16,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:16,791 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:16,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:16,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:16,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:16,791 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:16,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:16,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:16,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:16,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:16,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:16,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:16,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:16,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:16,798 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:16,798 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:16,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:16,898 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:16,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:16,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:16,902 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:16,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:16,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:16,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:16,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:16,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:16,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:16,904 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:16,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:16,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:16,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:16,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:16,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:16,915 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:16,916 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:16,916 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:16,934 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041324fb4dbc121444e7be9cb486f0db9f8e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fb4dbc121444e7be9cb486f0db9f8e 2023-04-13 02:55:16,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:16,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:16,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:16,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354576975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:17,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354577079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:17,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354577287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,354 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/d43d9987f96a4286a4d7c2d08f1c4a43 2023-04-13 02:55:17,365 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/de9704d004c249dab8d20305aa033b90 2023-04-13 02:55:17,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:17,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354577595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,781 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/1862433c510c4dac8c7f9218a8886b62 2023-04-13 02:55:17,786 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/d43d9987f96a4286a4d7c2d08f1c4a43 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43 2023-04-13 02:55:17,794 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43, entries=150, sequenceid=260, filesize=25.4 K 2023-04-13 02:55:17,794 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/de9704d004c249dab8d20305aa033b90 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90 2023-04-13 02:55:17,798 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90, entries=150, sequenceid=260, filesize=11.9 K 2023-04-13 02:55:17,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/1862433c510c4dac8c7f9218a8886b62 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62 2023-04-13 02:55:17,802 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62, entries=150, sequenceid=260, filesize=11.9 K 2023-04-13 02:55:17,803 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 888ms, sequenceid=260, compaction requested=true 2023-04-13 02:55:17,803 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:17,803 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:17,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:17,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:17,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:17,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc43c5b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:17,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:17,808 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:17,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:17,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:17,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:17,813 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:17,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:17,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:17,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:17,814 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:17,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:17,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,820 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:17,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:17,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:17,828 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:17,828 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:17,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:17,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:17,928 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:17,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:17,931 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:17,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:17,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:17,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:17,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:17,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:17,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:17,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:17,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:17,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:17,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:17,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:17,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:17,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:17,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:17,935 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:17,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:17,935 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:17,936 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:17,949 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136bfb371ffdcb4779ad0beef6df4e656f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bfb371ffdcb4779ad0beef6df4e656f 2023-04-13 02:55:17,958 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/0420c40c7e1c4162920f844a5e3f3db2 2023-04-13 02:55:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:18,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354518170, max timestamp=1681354518199Num uniques 859; 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354518170, max timestamp=1681354518199Num uniques 859; 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354518170, max timestamp=1681354518199Num uniques 859; 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:18,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:18,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354578295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354578331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,337 DEBUG [Thread-2138] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:55:18,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354578352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,357 DEBUG [Thread-2136] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:55:18,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354578353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354578354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,362 DEBUG [Thread-2142] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:55:18,363 DEBUG [Thread-2134] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:55:18,368 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/2889601688a941029244302184843914 2023-04-13 02:55:18,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354578399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354578604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,778 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/c7381dc96c354167bca6e432ce922d2b 2023-04-13 02:55:18,782 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/0420c40c7e1c4162920f844a5e3f3db2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2 2023-04-13 02:55:18,786 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2, entries=150, sequenceid=270, filesize=25.4 K 2023-04-13 02:55:18,786 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/2889601688a941029244302184843914 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914 2023-04-13 02:55:18,790 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914, entries=150, sequenceid=270, filesize=11.9 K 2023-04-13 02:55:18,791 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/c7381dc96c354167bca6e432ce922d2b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b 2023-04-13 02:55:18,794 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b, entries=150, sequenceid=270, filesize=11.9 K 2023-04-13 02:55:18,795 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 860ms, sequenceid=270, compaction requested=true 2023-04-13 02:55:18,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:18,795 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:18,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,797 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:18,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54dd5404[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:18,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:18,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:18,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:18,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:18,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:18,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:18,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:18,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:18,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:18,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:18,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:18,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:18,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:18,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,810 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:18,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:18,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:18,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:18,814 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,814 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:18,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,914 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:18,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:18,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:18,918 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:18,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:18,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:18,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:18,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-13 02:55:18,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:18,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:55:18,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:18,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:55:18,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:18,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:55:18,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:18,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:18,921 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:18,921 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,921 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:18,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:18,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:18,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:18,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:18,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:18,932 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:18,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:18,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad28f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:18,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:18,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:18,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:18,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:18,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:18,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:18,946 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:18,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:18,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:18,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:18,946 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:18,947 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413190b549972ef4ca7a3f956150da42c9f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413190b549972ef4ca7a3f956150da42c9f 2023-04-13 02:55:18,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:18,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:18,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:18,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:18,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:18,955 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:18,955 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:18,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:18,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354578973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,056 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,059 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,061 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,061 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d7ae11[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,070 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,073 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,073 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,074 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,080 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,080 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:19,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354579079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,180 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,184 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,189 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2d6254[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,195 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,199 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,199 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,201 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,201 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,201 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,205 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,208 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,207 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,208 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:19,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354579286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,308 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,311 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,313 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,314 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,318 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15ee7e41[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,321 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,321 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,322 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,323 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,325 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,325 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,326 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,326 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,326 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,331 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:19,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,332 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,332 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,354 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/6ed36dd1be864d5faff1bf5e2751bf7c 2023-04-13 02:55:19,432 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,437 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,444 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,447 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,447 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,449 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,452 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,452 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,452 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,452 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c528d9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,458 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,465 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,465 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,468 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,468 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:19,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,473 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,473 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,573 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,577 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,587 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,587 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,588 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,588 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,596 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa8b1f6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,596 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,597 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:19,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354579590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,599 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,599 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,604 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,604 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,606 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,607 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,607 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,609 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,612 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,611 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,612 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,612 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,612 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,712 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,715 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,715 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,715 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,715 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,716 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,717 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,718 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:19,719 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:19,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:19,721 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a574d35[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:19,722 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:19,722 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,724 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:19,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:19,724 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,725 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:19,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:19,726 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,726 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:19,726 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,726 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:19,729 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:19,729 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:19,729 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:19,729 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:19,729 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:19,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,731 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:19,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:19,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:19,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:19,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:19,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:19,735 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,735 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:19,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:19,735 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:19,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/e73f2643b4184127b70e363d4d0c0edb 2023-04-13 02:55:19,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/1b2976119dc6490a93bbdf4b2357f700 2023-04-13 02:55:19,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/6ed36dd1be864d5faff1bf5e2751bf7c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c 2023-04-13 02:55:19,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c, entries=200, sequenceid=308, filesize=32.3 K 2023-04-13 02:55:19,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/e73f2643b4184127b70e363d4d0c0edb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb 2023-04-13 02:55:19,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb, entries=150, sequenceid=308, filesize=11.9 K 2023-04-13 02:55:19,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/1b2976119dc6490a93bbdf4b2357f700 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700 2023-04-13 02:55:19,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700, entries=150, sequenceid=308, filesize=11.9 K 2023-04-13 02:55:19,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 906ms, sequenceid=308, compaction requested=true 2023-04-13 02:55:19,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:19,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:19,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:19,824 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:19,824 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:19,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:19,825 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60956 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:19,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137296 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:19,826 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/B is initiating minor compaction (all files) 2023-04-13 02:55:19,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/A is initiating minor compaction (all files) 2023-04-13 02:55:19,826 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/A in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,826 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b86eae5c4dd44b2aba26002d839fd401, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=134.1 K 2023-04-13 02:55:19,826 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/B in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,826 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bf9fb67759e14579b98157f7dcbc523e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=59.5 K 2023-04-13 02:55:19,826 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b86eae5c4dd44b2aba26002d839fd401, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:19,827 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting bf9fb67759e14579b98157f7dcbc523e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:19,827 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 107f922390c04f918f3229a71e7b2c41, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681354514180 2023-04-13 02:55:19,827 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting dd45578293a9484f8ce11487bebd4d79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681354514180 2023-04-13 02:55:19,827 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d43d9987f96a4286a4d7c2d08f1c4a43, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681354516283 2023-04-13 02:55:19,827 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting de9704d004c249dab8d20305aa033b90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681354516283 2023-04-13 02:55:19,828 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0420c40c7e1c4162920f844a5e3f3db2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681354516940 2023-04-13 02:55:19,828 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2889601688a941029244302184843914, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681354516940 2023-04-13 02:55:19,828 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6ed36dd1be864d5faff1bf5e2751bf7c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518262 2023-04-13 02:55:19,828 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e73f2643b4184127b70e363d4d0c0edb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518285 2023-04-13 02:55:19,836 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:19,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:19,850 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#B#compaction#328 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:19,857 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#A#compaction#327 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:19,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:19,860 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:19,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:19,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:19,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,863 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,863 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:19,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:19,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:19,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:19,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:19,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:19,882 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:19,882 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:19,882 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:19,902 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/83764c7fec9f4dfb9012a2407b046842 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/83764c7fec9f4dfb9012a2407b046842 2023-04-13 02:55:19,907 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/B of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 83764c7fec9f4dfb9012a2407b046842(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:19,907 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:19,907 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, priority=11, startTime=1681354519824; duration=0sec 2023-04-13 02:55:19,907 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:19,907 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 02:55:19,916 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60956 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 02:55:19,916 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/C is initiating minor compaction (all files) 2023-04-13 02:55:19,916 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/C in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:19,917 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/cbf5ba8779ca4fcb9b768721aaf716b9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=59.5 K 2023-04-13 02:55:19,917 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cbf5ba8779ca4fcb9b768721aaf716b9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681354513946 2023-04-13 02:55:19,918 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ec91e557ebe249638f98309e274749af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681354514180 2023-04-13 02:55:19,918 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1862433c510c4dac8c7f9218a8886b62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681354516283 2023-04-13 02:55:19,919 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c7381dc96c354167bca6e432ce922d2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681354516940 2023-04-13 02:55:19,920 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1b2976119dc6490a93bbdf4b2357f700, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518285 2023-04-13 02:55:19,921 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304130774235dc2274f688b140b2d283c891a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130774235dc2274f688b140b2d283c891a 2023-04-13 02:55:19,955 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#C#compaction#330 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:19,975 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/e8da736dc1c04205a70a235c4baec30b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e8da736dc1c04205a70a235c4baec30b 2023-04-13 02:55:19,979 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/C of 6ffe91e2c6ce88591f4f17c4b5e2f40a into e8da736dc1c04205a70a235c4baec30b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:19,979 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:19,979 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, priority=11, startTime=1681354519824; duration=0sec 2023-04-13 02:55:19,979 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:20,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:20,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354520176, max timestamp=1681354520259Num uniques 859; 2023-04-13 02:55:20,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:20,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354580311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,348 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/2e2f89ab426a4701b68b5e2dfa41344a 2023-04-13 02:55:20,366 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/0b64ca0bd4b34b008a8f8787fe82f545 2023-04-13 02:55:20,382 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/ea52be818b1045e0aa84776702512c22 2023-04-13 02:55:20,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/2e2f89ab426a4701b68b5e2dfa41344a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a 2023-04-13 02:55:20,392 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a, entries=150, sequenceid=317, filesize=25.4 K 2023-04-13 02:55:20,393 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/0b64ca0bd4b34b008a8f8787fe82f545 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545 2023-04-13 02:55:20,401 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545, entries=150, sequenceid=317, filesize=11.9 K 2023-04-13 02:55:20,402 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/ea52be818b1045e0aa84776702512c22 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22 2023-04-13 02:55:20,407 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22, entries=150, sequenceid=317, filesize=11.9 K 2023-04-13 02:55:20,408 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=136.42 KB/139690 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 526ms, sequenceid=317, compaction requested=false 2023-04-13 02:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:20,408 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:20,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2d3e48[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:20,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:20,412 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,413 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:20,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:20,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:20,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,415 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:20,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:20,421 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:20,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:20,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:20,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:20,421 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:20,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,425 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:20,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=143.13 KB heapSize=421.48 KB 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:20,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:20,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:20,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:20,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,432 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:20,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,432 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,433 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:20,443 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041306eac09208344174bbf334728481e3c3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306eac09208344174bbf334728481e3c3 2023-04-13 02:55:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354580469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,533 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:20,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:20,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:20,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:20,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:20,537 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:20,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:20,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:20,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:20,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:20,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:20,540 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,540 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:20,540 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:20,540 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:20,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:20,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454748f4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:20,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:20,552 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,553 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:20,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:20,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:20,557 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:20,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:20,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:20,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:20,557 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:20,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,566 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,567 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,569 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:20,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:20,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,570 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:20,570 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:20,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354580582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,670 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:20,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:20,673 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:20,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:20,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:20,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:20,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:20,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:20,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:20,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:20,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:20,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:20,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:20,680 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:20,680 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2f66ab[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:20,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:20,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:20,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:20,688 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:20,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:20,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:20,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:20,688 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:20,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:20,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:20,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,695 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:20,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,695 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,712 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/466e0f5378f74aecbce64c40fbed1a25 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/466e0f5378f74aecbce64c40fbed1a25 2023-04-13 02:55:20,716 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/A of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 466e0f5378f74aecbce64c40fbed1a25(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:20,716 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:20,716 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, priority=11, startTime=1681354519824; duration=0sec 2023-04-13 02:55:20,716 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:20,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354580786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,795 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:20,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:20,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:20,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:20,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:20,802 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:20,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:20,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:20,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:20,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:20,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:20,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:20,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:20,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:20,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:20,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:20,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a61bc3e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:20,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:20,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:20,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:20,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:20,815 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,815 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:20,819 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:20,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:20,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:20,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:20,819 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:20,821 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,822 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:20,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:20,826 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:20,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:20,826 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:20,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,848 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/0645718816964526911779c42609ba8e 2023-04-13 02:55:20,926 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:20,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:20,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:20,929 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:20,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:20,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:20,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:20,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:20,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:20,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:20,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:20,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:20,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:20,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:20,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:20,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7b1244[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:20,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:20,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:20,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:20,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:20,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:20,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,946 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:20,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:20,949 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:20,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:20,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:20,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:20,949 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:20,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,953 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:20,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:20,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:20,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:20,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:20,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:20,955 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:20,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:20,955 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:20,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,055 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:21,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:21,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:21,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:21,058 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:21,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:21,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:21,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,061 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:21,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:21,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:21,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:21,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:21,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:21,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:21,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:21,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:21,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:21,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:21,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4004edb7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:21,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:21,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:21,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:21,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:21,072 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:21,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:21,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:21,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:21,072 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:21,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:21,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:21,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:21,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:21,079 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:21,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,079 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:21,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354581092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,180 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:21,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:21,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:21,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:21,183 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:21,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,183 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:21,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:21,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,184 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:21,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:21,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:21,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:21,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:21,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:21,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:21,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:21,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:21,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:21,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:21,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:21,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:21,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:21,191 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316fd122[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:21,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:21,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:21,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:21,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:21,207 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:21,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:21,207 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:21,209 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,209 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:21,209 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:21,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:21,210 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:21,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:21,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:21,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:21,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:21,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:21,213 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:21,214 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:21,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,214 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:21,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/6a8bff81cf374996ae29621ee629e180 2023-04-13 02:55:21,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/8df2adf45763485f832cd29d6406e41f 2023-04-13 02:55:21,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/0645718816964526911779c42609ba8e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e 2023-04-13 02:55:21,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e, entries=200, sequenceid=348, filesize=32.3 K 2023-04-13 02:55:21,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/6a8bff81cf374996ae29621ee629e180 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180 2023-04-13 02:55:21,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180, entries=150, sequenceid=348, filesize=11.9 K 2023-04-13 02:55:21,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/8df2adf45763485f832cd29d6406e41f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f 2023-04-13 02:55:21,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f, entries=150, sequenceid=348, filesize=11.9 K 2023-04-13 02:55:21,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~421.44 KB/431552, currentSize=33.54 KB/34350 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 883ms, sequenceid=348, compaction requested=true 2023-04-13 02:55:21,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:21,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:21,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:21,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:21,309 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:21,309 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:21,314 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:21,315 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:21,315 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/B is initiating minor compaction (all files) 2023-04-13 02:55:21,315 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85724 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:21,315 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/B in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,315 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/A is initiating minor compaction (all files) 2023-04-13 02:55:21,316 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/A in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:21,316 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/83764c7fec9f4dfb9012a2407b046842, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=36.2 K 2023-04-13 02:55:21,316 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/466e0f5378f74aecbce64c40fbed1a25, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=83.7 K 2023-04-13 02:55:21,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:21,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 466e0f5378f74aecbce64c40fbed1a25, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518285 2023-04-13 02:55:21,319 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 83764c7fec9f4dfb9012a2407b046842, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518285 2023-04-13 02:55:21,324 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0b64ca0bd4b34b008a8f8787fe82f545, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681354518928 2023-04-13 02:55:21,324 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2e2f89ab426a4701b68b5e2dfa41344a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681354518928 2023-04-13 02:55:21,324 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 6a8bff81cf374996ae29621ee629e180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520306 2023-04-13 02:55:21,325 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0645718816964526911779c42609ba8e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520278 2023-04-13 02:55:21,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:21,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:21,325 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:21,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:21,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:21,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:21,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:21,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:21,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:21,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:21,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:21,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:21,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:21,330 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,330 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:21,330 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 02:55:21,330 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:21,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:21,332 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#B#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:21,354 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#A#compaction#337 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:21,410 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413219a0948e6b94e039265f5de19cbb0dc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413219a0948e6b94e039265f5de19cbb0dc 2023-04-13 02:55:21,443 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/4427b11dadac4ea6860f160aee7f432a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4427b11dadac4ea6860f160aee7f432a 2023-04-13 02:55:21,448 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/A of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 4427b11dadac4ea6860f160aee7f432a(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:21,448 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:21,448 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, priority=13, startTime=1681354521309; duration=0sec 2023-04-13 02:55:21,448 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:21,448 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:21,449 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:21,449 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/C is initiating minor compaction (all files) 2023-04-13 02:55:21,450 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/C in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:21,450 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e8da736dc1c04205a70a235c4baec30b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=36.2 K 2023-04-13 02:55:21,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e8da736dc1c04205a70a235c4baec30b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681354518285 2023-04-13 02:55:21,450 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ea52be818b1045e0aa84776702512c22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681354518928 2023-04-13 02:55:21,451 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8df2adf45763485f832cd29d6406e41f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520306 2023-04-13 02:55:21,458 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:21,474 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/e9396c9909e24b4c8fa9e16fea16c519 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e9396c9909e24b4c8fa9e16fea16c519 2023-04-13 02:55:21,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/C of 6ffe91e2c6ce88591f4f17c4b5e2f40a into e9396c9909e24b4c8fa9e16fea16c519(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:21,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:21,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, priority=13, startTime=1681354521309; duration=0sec 2023-04-13 02:55:21,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:21,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:21,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:21,754 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/3d0cd3cb2b354c8cb159a0dc91d9a437 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3d0cd3cb2b354c8cb159a0dc91d9a437 2023-04-13 02:55:21,763 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/B of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 3d0cd3cb2b354c8cb159a0dc91d9a437(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:21,763 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:21,763 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, priority=13, startTime=1681354521309; duration=0sec 2023-04-13 02:55:21,763 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354521746, max timestamp=1681354521786Num uniques 859; 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354521746, max timestamp=1681354521786Num uniques 859; 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354521746, max timestamp=1681354521786Num uniques 859; 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:21,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:21,834 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/3e095d3459f948b380057d173d94b628 2023-04-13 02:55:21,875 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/238f30fbb93d40b6bafb7f685ac7a75b 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354521877, max timestamp=1681354521895Num uniques 1018; 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354521877, max timestamp=1681354521895Num uniques 1018; 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354521877, max timestamp=1681354521895Num uniques 1018; 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:55:21,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:55:22,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681354521977, max timestamp=1681354522017Num uniques 1027; 2023-04-13 02:55:22,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 02:55:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681354521977, max timestamp=1681354522017Num uniques 1027; 2023-04-13 02:55:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 02:55:22,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354521977, max timestamp=1681354522017Num uniques 1027; 2023-04-13 02:55:22,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-13 02:55:22,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681354522085, max timestamp=1681354522112Num uniques 1036; 2023-04-13 02:55:22,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681354522085, max timestamp=1681354522112Num uniques 1036; 2023-04-13 02:55:22,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 02:55:22,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 02:55:22,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681354522088, max timestamp=1681354522112Num uniques 1009; 2023-04-13 02:55:22,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-13 02:55:22,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354522184, max timestamp=1681354522213Num uniques 1095; 2023-04-13 02:55:22,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-13 02:55:22,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354522184, max timestamp=1681354522213Num uniques 1045; 2023-04-13 02:55:22,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 02:55:22,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681354522184, max timestamp=1681354522213Num uniques 1095; 2023-04-13 02:55:22,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-13 02:55:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681354522292, max timestamp=1681354522314Num uniques 1054; 2023-04-13 02:55:22,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 02:55:22,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681354522292, max timestamp=1681354522314Num uniques 1054; 2023-04-13 02:55:22,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 02:55:22,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681354522292, max timestamp=1681354522314Num uniques 1054; 2023-04-13 02:55:22,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-13 02:55:22,318 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/7f6e9eea31324030ab80e4a35b20a025 2023-04-13 02:55:22,323 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/3e095d3459f948b380057d173d94b628 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628 2023-04-13 02:55:22,327 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628, entries=100, sequenceid=357, filesize=18.6 K 2023-04-13 02:55:22,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/238f30fbb93d40b6bafb7f685ac7a75b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b 2023-04-13 02:55:22,336 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b, entries=100, sequenceid=357, filesize=9.5 K 2023-04-13 02:55:22,338 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/7f6e9eea31324030ab80e4a35b20a025 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025 2023-04-13 02:55:22,341 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025, entries=100, sequenceid=357, filesize=9.5 K 2023-04-13 02:55:22,342 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=60.38 KB/61830 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1011ms, sequenceid=357, compaction requested=false 2023-04-13 02:55:22,342 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:22,342 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:22,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,344 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,345 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,345 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46980b7d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:22,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:22,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=67.09 KB heapSize=181.55 KB 2023-04-13 02:55:22,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:22,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:22,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:22,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:22,348 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:22,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:22,351 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,351 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:22,352 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,352 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:22,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:22,354 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:22,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:22,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:22,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:22,354 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:22,358 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,358 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,359 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,362 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,364 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:22,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:22,365 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:22,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:22,365 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,366 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,391 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413afdafe144beb45f1b93e907afeed8488 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afdafe144beb45f1b93e907afeed8488 2023-04-13 02:55:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354582410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354522408, max timestamp=1681354522418Num uniques 859; 2023-04-13 02:55:22,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354522408, max timestamp=1681354522418Num uniques 859; 2023-04-13 02:55:22,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:22,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:22,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354522408, max timestamp=1681354522418Num uniques 859; 2023-04-13 02:55:22,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:22,425 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=472, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/d7e5542b421843b9ae4f5433e8e53424 2023-04-13 02:55:22,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/299fc57c51ab4c37bda7e1b865382a14 2023-04-13 02:55:22,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/4f1d9304e68d431191c9cc2ef8f7441c 2023-04-13 02:55:22,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/d7e5542b421843b9ae4f5433e8e53424 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424 2023-04-13 02:55:22,465 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:22,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:22,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424, entries=200, sequenceid=472, filesize=32.3 K 2023-04-13 02:55:22,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/299fc57c51ab4c37bda7e1b865382a14 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14 2023-04-13 02:55:22,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354582457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:22,470 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:22,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:22,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:22,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14, entries=150, sequenceid=472, filesize=11.9 K 2023-04-13 02:55:22,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/4f1d9304e68d431191c9cc2ef8f7441c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c 2023-04-13 02:55:22,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354582460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,477 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:22,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:22,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354582463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354582464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:22,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:22,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:22,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:22,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:22,489 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,489 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:22,489 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:22,489 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:22,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,492 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c, entries=150, sequenceid=472, filesize=11.9 K 2023-04-13 02:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2116c8d8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:22,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:22,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:22,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:22,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:22,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:22,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.50 KB/185856, currentSize=120.76 KB/123660 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 154ms, sequenceid=472, compaction requested=true 2023-04-13 02:55:22,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:22,501 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:22,501 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:22,502 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,503 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:22,503 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/B is initiating minor compaction (all files) 2023-04-13 02:55:22,503 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/B in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,503 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78778 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:22,503 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/A is initiating minor compaction (all files) 2023-04-13 02:55:22,503 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3d0cd3cb2b354c8cb159a0dc91d9a437, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=33.9 K 2023-04-13 02:55:22,503 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/A in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,503 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4427b11dadac4ea6860f160aee7f432a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=76.9 K 2023-04-13 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,505 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4427b11dadac4ea6860f160aee7f432a, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520306 2023-04-13 02:55:22,505 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3d0cd3cb2b354c8cb159a0dc91d9a437, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520306 2023-04-13 02:55:22,505 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3e095d3459f948b380057d173d94b628, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681354520463 2023-04-13 02:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:22,506 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:22,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:22,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:22,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:22,506 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:22,507 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d7e5542b421843b9ae4f5433e8e53424, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681354522325 2023-04-13 02:55:22,507 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 238f30fbb93d40b6bafb7f685ac7a75b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681354520463 2023-04-13 02:55:22,508 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,508 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,508 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,508 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 299fc57c51ab4c37bda7e1b865382a14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681354522325 2023-04-13 02:55:22,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,511 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:22,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:22,516 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,517 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,517 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:22,517 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:22,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:22,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:55:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:22,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:22,529 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#B#compaction#346 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:22,562 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#A#compaction#345 average throughput is 0.48 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:22,605 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/bd01622192054a50a35f9da3d3b5f9bb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bd01622192054a50a35f9da3d3b5f9bb 2023-04-13 02:55:22,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354582585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,616 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/B of 6ffe91e2c6ce88591f4f17c4b5e2f40a into bd01622192054a50a35f9da3d3b5f9bb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:22,616 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:22,616 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/B, priority=13, startTime=1681354522501; duration=0sec 2023-04-13 02:55:22,616 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:22,616 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:22,617 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:22,617 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:22,617 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 6ffe91e2c6ce88591f4f17c4b5e2f40a/C is initiating minor compaction (all files) 2023-04-13 02:55:22,618 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6ffe91e2c6ce88591f4f17c4b5e2f40a/C in TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,618 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e9396c9909e24b4c8fa9e16fea16c519, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp, totalSize=33.9 K 2023-04-13 02:55:22,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:22,618 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e9396c9909e24b4c8fa9e16fea16c519, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681354520306 2023-04-13 02:55:22,619 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7f6e9eea31324030ab80e4a35b20a025, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681354520463 2023-04-13 02:55:22,619 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4f1d9304e68d431191c9cc2ef8f7441c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1681354522325 2023-04-13 02:55:22,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:22,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354582602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354582604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354582609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354582616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:22,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:22,632 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:22,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:22,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:22,634 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 6ffe91e2c6ce88591f4f17c4b5e2f40a#C#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:22,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,642 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:22,645 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/233ed51e53104d999ba35c81616cfb94 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/233ed51e53104d999ba35c81616cfb94 2023-04-13 02:55:22,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:22,652 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/A of 6ffe91e2c6ce88591f4f17c4b5e2f40a into 233ed51e53104d999ba35c81616cfb94(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:22,652 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:22,652 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/A, priority=13, startTime=1681354522501; duration=0sec 2023-04-13 02:55:22,652 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:22,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:22,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:22,659 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,659 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:22,659 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:22,659 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:22,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:22,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:22,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4698e018[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:22,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:22,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,674 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:22,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,674 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:22,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:22,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:22,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:22,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:22,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:22,678 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:22,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:22,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:22,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,689 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:22,689 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:22,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354582724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354582726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354582729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354582730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,789 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:22,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:22,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:22,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:22,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:22,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:22,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:22,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:22,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:22,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:22,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:22,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:22,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:22,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b1cf28[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:22,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:22,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:22,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:22,812 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:22,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:22,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:22,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:22,812 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:22,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354582811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,824 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:22,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:22,824 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:22,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:22,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,824 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,925 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:22,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:22,928 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:22,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:22,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:22,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:22,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:22,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f74fe80[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:22,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:22,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354582934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41166 deadline: 1681354582936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:22,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:22,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:22,942 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354582939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:22,946 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:22,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:22,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:22,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:22,946 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:22,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:22,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354582940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:22,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:22,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:22,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:22,956 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:22,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:22,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:22,956 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:22,956 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,005 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f5ef376ff9754bc8b6e92f36d4061a31 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5ef376ff9754bc8b6e92f36d4061a31 2023-04-13 02:55:23,056 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,064 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,073 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/f0000277ec00409ca06d1079db66819b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f0000277ec00409ca06d1079db66819b 2023-04-13 02:55:23,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,074 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,075 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,075 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,075 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,077 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,079 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6ffe91e2c6ce88591f4f17c4b5e2f40a/C of 6ffe91e2c6ce88591f4f17c4b5e2f40a into f0000277ec00409ca06d1079db66819b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:23,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:23,079 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a., storeName=6ffe91e2c6ce88591f4f17c4b5e2f40a/C, priority=13, startTime=1681354522501; duration=0sec 2023-04-13 02:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1833b7be[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,084 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,084 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,084 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,084 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,085 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,086 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,088 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,091 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,091 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,091 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,091 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,091 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,091 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354583119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,191 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,195 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,198 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,201 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,201 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,201 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,201 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192a1c80[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,207 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,208 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,210 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,210 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,210 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,211 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,211 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,212 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,212 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,212 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,220 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,220 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,221 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,223 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,223 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1681354583239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354583240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354583247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354583251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,323 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,326 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,330 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,330 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,334 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ee55802[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,335 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,336 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,337 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,338 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,338 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,338 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,339 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,341 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,341 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,341 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,341 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,342 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,345 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,348 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,348 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,411 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=507, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a9a158086e754ec19b6b03e25e6e1315 2023-04-13 02:55:23,449 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,453 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,457 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@364d25a0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,458 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,465 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,466 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,466 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,468 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,468 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,468 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,468 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,468 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,474 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,474 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,474 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,474 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,474 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,474 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,474 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,574 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,580 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,583 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,587 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c6ea90[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,590 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,590 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,591 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,591 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,593 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,598 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,598 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,602 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,603 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,603 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,604 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,608 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:23,608 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,608 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,608 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,609 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,609 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41162 deadline: 1681354583625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,708 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,712 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,712 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,713 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,714 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,714 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,715 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,715 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,716 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,716 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,720 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,721 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a413e74[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,722 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,722 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,723 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,723 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,724 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,725 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,725 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,727 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,727 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,727 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,729 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,731 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,733 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,734 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:23,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,734 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,734 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,734 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,734 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41196 deadline: 1681354583746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41166 deadline: 1681354583749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41178 deadline: 1681354583760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41182 deadline: 1681354583760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/c45f73218dd842638f59dede408ec14d 2023-04-13 02:55:23,834 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,837 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,838 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,839 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,839 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,839 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,839 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e505c76[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,845 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,847 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,851 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,851 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,851 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,854 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:23,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,857 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,857 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,957 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:23,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:23,960 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:23,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:23,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:23,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,963 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,963 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:23,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:23,967 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,967 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:23,967 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:23,967 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:23,967 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:23,969 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,969 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,970 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116f1039[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:23,971 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:23,971 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:23,973 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,973 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:23,974 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:23,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:23,977 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:23,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:23,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:23,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:23,977 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:23,979 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,979 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,979 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:23,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:23,980 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,981 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:23,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:23,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:23,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:23,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:23,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:23,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:23,985 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:23,986 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:23,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:23,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:23,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,086 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:24,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:24,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:24,089 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:24,089 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:24,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:24,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:24,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:24,092 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,092 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:24,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:24,092 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,092 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:24,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:24,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:24,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:24,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:24,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:24,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:24,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:24,094 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:24,094 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. started... 2023-04-13 02:55:24,094 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. as already flushing 2023-04-13 02:55:24,094 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:24,094 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:24,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:24,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f9eaee[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:24,097 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,098 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:24,099 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:24,099 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,099 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:24,099 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:24,104 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:24,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:24,106 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:24,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:24,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:24,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:24,106 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:24,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:24,108 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:24,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,112 DEBUG [Thread-2149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x750d2a7d to 127.0.0.1:62948 2023-04-13 02:55:24,112 DEBUG [Thread-2149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,112 DEBUG [Thread-2151] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27313b38 to 127.0.0.1:62948 2023-04-13 02:55:24,113 DEBUG [Thread-2151] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:24,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:24,114 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:24,114 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:24,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:24,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:24,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:24,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:24,118 DEBUG [Thread-2144] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:24,118 DEBUG [Thread-2144] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:24,121 DEBUG [Thread-2147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20d28b68 to 127.0.0.1:62948 2023-04-13 02:55:24,122 DEBUG [Thread-2147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,122 DEBUG [Thread-2153] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ee4c750 to 127.0.0.1:62948 2023-04-13 02:55:24,122 DEBUG [Thread-2153] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,123 DEBUG [Thread-2145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x313c7ad6 to 127.0.0.1:62948 2023-04-13 02:55:24,123 DEBUG [Thread-2145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,218 DEBUG [Thread-2144] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:24,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:24,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/9b402c7fd182459bae5454224be25b9d 2023-04-13 02:55:24,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a9a158086e754ec19b6b03e25e6e1315 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a9a158086e754ec19b6b03e25e6e1315 2023-04-13 02:55:24,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a9a158086e754ec19b6b03e25e6e1315, entries=200, sequenceid=507, filesize=32.3 K 2023-04-13 02:55:24,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/c45f73218dd842638f59dede408ec14d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/c45f73218dd842638f59dede408ec14d 2023-04-13 02:55:24,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/c45f73218dd842638f59dede408ec14d, entries=150, sequenceid=507, filesize=11.9 K 2023-04-13 02:55:24,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/9b402c7fd182459bae5454224be25b9d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9b402c7fd182459bae5454224be25b9d 2023-04-13 02:55:24,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9b402c7fd182459bae5454224be25b9d, entries=150, sequenceid=507, filesize=11.9 K 2023-04-13 02:55:24,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1717ms, sequenceid=507, compaction requested=false 2023-04-13 02:55:24,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:24,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:55:24,632 DEBUG [Thread-2136] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14e151fb to 127.0.0.1:62948 2023-04-13 02:55:24,633 DEBUG [Thread-2136] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:24,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:24,650 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413425881324f2d435296d8175136f113b8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413425881324f2d435296d8175136f113b8 2023-04-13 02:55:24,759 DEBUG [Thread-2134] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c1d19b3 to 127.0.0.1:62948 2023-04-13 02:55:24,759 DEBUG [Thread-2134] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,761 DEBUG [Thread-2142] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x066bf91c to 127.0.0.1:62948 2023-04-13 02:55:24,761 DEBUG [Thread-2142] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,767 DEBUG [Thread-2138] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x455a88b1 to 127.0.0.1:62948 2023-04-13 02:55:24,767 DEBUG [Thread-2138] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,772 DEBUG [Thread-2140] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x722b2407 to 127.0.0.1:62948 2023-04-13 02:55:24,772 DEBUG [Thread-2140] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 291 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1887 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5660 rows 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1900 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5699 rows 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1886 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5658 rows 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1884 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5652 rows 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1893 2023-04-13 02:55:24,773 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5679 rows 2023-04-13 02:55:24,774 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:55:24,774 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cee66b5 to 127.0.0.1:62948 2023-04-13 02:55:24,774 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:24,777 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:55:24,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:55:24,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:24,781 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354524781"}]},"ts":"1681354524781"} 2023-04-13 02:55:24,782 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:55:24,879 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:55:24,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, UNASSIGN}] 2023-04-13 02:55:24,881 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, UNASSIGN 2023-04-13 02:55:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:24,882 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:24,882 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354524882"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354524882"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354524882"}]},"ts":"1681354524882"} 2023-04-13 02:55:24,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:25,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:25,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6ffe91e2c6ce88591f4f17c4b5e2f40a, disabling compactions & flushes 2023-04-13 02:55:25,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:25,055 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=523, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a5055e1164924d6483abb0dcd2581e08 2023-04-13 02:55:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:25,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/3e625b11553e4174ae9aa6af680e9827 2023-04-13 02:55:25,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/24e12734d48647ec8c6cd6786f9e7695 2023-04-13 02:55:25,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/a5055e1164924d6483abb0dcd2581e08 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a5055e1164924d6483abb0dcd2581e08 2023-04-13 02:55:25,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a5055e1164924d6483abb0dcd2581e08, entries=150, sequenceid=523, filesize=25.4 K 2023-04-13 02:55:25,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/3e625b11553e4174ae9aa6af680e9827 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3e625b11553e4174ae9aa6af680e9827 2023-04-13 02:55:25,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3e625b11553e4174ae9aa6af680e9827, entries=150, sequenceid=523, filesize=11.9 K 2023-04-13 02:55:25,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/24e12734d48647ec8c6cd6786f9e7695 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/24e12734d48647ec8c6cd6786f9e7695 2023-04-13 02:55:25,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:25,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/24e12734d48647ec8c6cd6786f9e7695, entries=150, sequenceid=523, filesize=11.9 K 2023-04-13 02:55:25,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1256ms, sequenceid=523, compaction requested=true 2023-04-13 02:55:25,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:25,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:25,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:25,888 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. because compaction request was cancelled 2023-04-13 02:55:25,888 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. because compaction request was cancelled 2023-04-13 02:55:25,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:25,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:25,888 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. because compaction request was cancelled 2023-04-13 02:55:25,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:25,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. after waiting 0 ms 2023-04-13 02:55:25,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:25,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6ffe91e2c6ce88591f4f17c4b5e2f40a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=A 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=B 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6ffe91e2c6ce88591f4f17c4b5e2f40a, store=C 2023-04-13 02:55:25,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:26,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136afb34c344854bec899bd8791810f832 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136afb34c344854bec899bd8791810f832 2023-04-13 02:55:26,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=530, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/5733bbea979948aaa185c3ae096c7d0d 2023-04-13 02:55:26,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/5fc0910a4fbf4da2b3ff454ae180ab3b 2023-04-13 02:55:26,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:27,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/42d89805131a40598af2febbb04166dc 2023-04-13 02:55:27,130 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/A/5733bbea979948aaa185c3ae096c7d0d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5733bbea979948aaa185c3ae096c7d0d 2023-04-13 02:55:27,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5733bbea979948aaa185c3ae096c7d0d, entries=150, sequenceid=530, filesize=25.4 K 2023-04-13 02:55:27,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/B/5fc0910a4fbf4da2b3ff454ae180ab3b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/5fc0910a4fbf4da2b3ff454ae180ab3b 2023-04-13 02:55:27,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/5fc0910a4fbf4da2b3ff454ae180ab3b, entries=150, sequenceid=530, filesize=11.9 K 2023-04-13 02:55:27,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/.tmp/C/42d89805131a40598af2febbb04166dc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/42d89805131a40598af2febbb04166dc 2023-04-13 02:55:27,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/42d89805131a40598af2febbb04166dc, entries=150, sequenceid=530, filesize=11.9 K 2023-04-13 02:55:27,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6ffe91e2c6ce88591f4f17c4b5e2f40a in 1253ms, sequenceid=530, compaction requested=true 2023-04-13 02:55:27,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fb1402df26af49228459c4e88fa10862, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b86eae5c4dd44b2aba26002d839fd401, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/466e0f5378f74aecbce64c40fbed1a25, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4427b11dadac4ea6860f160aee7f432a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424] to archive 2023-04-13 02:55:27,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:27,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5851df8f9eed482a92234bb6cb893020 2023-04-13 02:55:27,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4fc3abaac5234c2ab6422a693921d919 2023-04-13 02:55:27,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2a5b87b54d4240178972a77da12209eb 2023-04-13 02:55:27,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a0cc7c371c3643f9a05f9134cbc196f2 2023-04-13 02:55:27,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/7162ec7bcd9f47b8b44eb8c35f191664 2023-04-13 02:55:27,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fb1402df26af49228459c4e88fa10862 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fb1402df26af49228459c4e88fa10862 2023-04-13 02:55:27,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/cab9cec650c04b3089bcb914f7c24028 2023-04-13 02:55:27,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/546e279f90b1417ab89d18fa7163aaef 2023-04-13 02:55:27,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/fa0ad743565c4260a327e46d55a67849 2023-04-13 02:55:27,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b99ab9c70b9743f88f98802122dae80a 2023-04-13 02:55:27,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ca3f08c255e445c94432281abcefc86 2023-04-13 02:55:27,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b86eae5c4dd44b2aba26002d839fd401 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/b86eae5c4dd44b2aba26002d839fd401 2023-04-13 02:55:27,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/107f922390c04f918f3229a71e7b2c41 2023-04-13 02:55:27,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d43d9987f96a4286a4d7c2d08f1c4a43 2023-04-13 02:55:27,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0420c40c7e1c4162920f844a5e3f3db2 2023-04-13 02:55:27,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/6ed36dd1be864d5faff1bf5e2751bf7c 2023-04-13 02:55:27,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/466e0f5378f74aecbce64c40fbed1a25 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/466e0f5378f74aecbce64c40fbed1a25 2023-04-13 02:55:27,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/2e2f89ab426a4701b68b5e2dfa41344a 2023-04-13 02:55:27,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/0645718816964526911779c42609ba8e 2023-04-13 02:55:27,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4427b11dadac4ea6860f160aee7f432a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/4427b11dadac4ea6860f160aee7f432a 2023-04-13 02:55:27,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/3e095d3459f948b380057d173d94b628 2023-04-13 02:55:27,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/d7e5542b421843b9ae4f5433e8e53424 2023-04-13 02:55:27,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/93862df34c7c4bd38c6ecebaaa296a75, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bf9fb67759e14579b98157f7dcbc523e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/83764c7fec9f4dfb9012a2407b046842, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3d0cd3cb2b354c8cb159a0dc91d9a437, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14] to archive 2023-04-13 02:55:27,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:27,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/f2e5806321264baa9cfa525e3d3d4ff7 2023-04-13 02:55:27,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/303b3568f2354255a7c967eb77d5c886 2023-04-13 02:55:27,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/48ca9f1a8589473cb53efa8de84970b7 2023-04-13 02:55:27,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/462774ca7e2a417ab8fbb1e37bebc5cb 2023-04-13 02:55:27,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/93862df34c7c4bd38c6ecebaaa296a75 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/93862df34c7c4bd38c6ecebaaa296a75 2023-04-13 02:55:27,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/98ad2b297f794837950754b2d5cbe642 2023-04-13 02:55:27,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de8c4f2f056d462288bcca720a1b9808 2023-04-13 02:55:27,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/d4fb4dc01444489ab6af742b7b1a862b 2023-04-13 02:55:27,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/38f20c9af1844cea9b804d330097f5e0 2023-04-13 02:55:27,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/249d62fe54be4358bf08817dff428afe 2023-04-13 02:55:27,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bf9fb67759e14579b98157f7dcbc523e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bf9fb67759e14579b98157f7dcbc523e 2023-04-13 02:55:27,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/7836b53b5df14459937a2b11301f2451 2023-04-13 02:55:27,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/dd45578293a9484f8ce11487bebd4d79 2023-04-13 02:55:27,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/de9704d004c249dab8d20305aa033b90 2023-04-13 02:55:27,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/2889601688a941029244302184843914 2023-04-13 02:55:27,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/83764c7fec9f4dfb9012a2407b046842 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/83764c7fec9f4dfb9012a2407b046842 2023-04-13 02:55:27,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/e73f2643b4184127b70e363d4d0c0edb 2023-04-13 02:55:27,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/0b64ca0bd4b34b008a8f8787fe82f545 2023-04-13 02:55:27,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3d0cd3cb2b354c8cb159a0dc91d9a437 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3d0cd3cb2b354c8cb159a0dc91d9a437 2023-04-13 02:55:27,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/6a8bff81cf374996ae29621ee629e180 2023-04-13 02:55:27,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/238f30fbb93d40b6bafb7f685ac7a75b 2023-04-13 02:55:27,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/299fc57c51ab4c37bda7e1b865382a14 2023-04-13 02:55:27,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a9c67f8e26064eceb2c3394d954d2915, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/cbf5ba8779ca4fcb9b768721aaf716b9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e8da736dc1c04205a70a235c4baec30b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e9396c9909e24b4c8fa9e16fea16c519, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c] to archive 2023-04-13 02:55:27,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:27,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/69eaf52f8ba548b187770f9f40a9fbb7 2023-04-13 02:55:27,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/10afe8fa47ef4bfcb0a3623c89a39e54 2023-04-13 02:55:27,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a92afd18d4754cb18bae5c8415c56015 2023-04-13 02:55:27,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9fef76d50aec4b7dbfc58f7a91c9ad1c 2023-04-13 02:55:27,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a9c67f8e26064eceb2c3394d954d2915 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a9c67f8e26064eceb2c3394d954d2915 2023-04-13 02:55:27,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/65de4f418e634fde863a1684d8821642 2023-04-13 02:55:27,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c91c5f686dbf4aaca429153446b1ead0 2023-04-13 02:55:27,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/a5a6c9a6a4154e2ca9f16196f48a5399 2023-04-13 02:55:27,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f6cb313a488f41bea4c8e30d0c026470 2023-04-13 02:55:27,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7653d945faa249cba3f4f5f0d296da51 2023-04-13 02:55:27,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/cbf5ba8779ca4fcb9b768721aaf716b9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/cbf5ba8779ca4fcb9b768721aaf716b9 2023-04-13 02:55:27,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f7666edcb93f41048c972b1ec20fcd28 2023-04-13 02:55:27,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ec91e557ebe249638f98309e274749af 2023-04-13 02:55:27,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1862433c510c4dac8c7f9218a8886b62 2023-04-13 02:55:27,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/c7381dc96c354167bca6e432ce922d2b 2023-04-13 02:55:27,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e8da736dc1c04205a70a235c4baec30b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e8da736dc1c04205a70a235c4baec30b 2023-04-13 02:55:27,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/1b2976119dc6490a93bbdf4b2357f700 2023-04-13 02:55:27,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/ea52be818b1045e0aa84776702512c22 2023-04-13 02:55:27,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e9396c9909e24b4c8fa9e16fea16c519 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/e9396c9909e24b4c8fa9e16fea16c519 2023-04-13 02:55:27,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/8df2adf45763485f832cd29d6406e41f 2023-04-13 02:55:27,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/7f6e9eea31324030ab80e4a35b20a025 2023-04-13 02:55:27,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/4f1d9304e68d431191c9cc2ef8f7441c 2023-04-13 02:55:27,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits/533.seqid, newMaxSeqId=533, maxSeqId=4 2023-04-13 02:55:27,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a. 2023-04-13 02:55:27,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6ffe91e2c6ce88591f4f17c4b5e2f40a: 2023-04-13 02:55:27,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:27,214 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=6ffe91e2c6ce88591f4f17c4b5e2f40a, regionState=CLOSED 2023-04-13 02:55:27,214 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354527214"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354527214"}]},"ts":"1681354527214"} 2023-04-13 02:55:27,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-13 02:55:27,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 6ffe91e2c6ce88591f4f17c4b5e2f40a, server=jenkins-hbase4.apache.org,34919,1681354416534 in 2.3320 sec 2023-04-13 02:55:27,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-13 02:55:27,217 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6ffe91e2c6ce88591f4f17c4b5e2f40a, UNASSIGN in 2.3370 sec 2023-04-13 02:55:27,218 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354527218"}]},"ts":"1681354527218"} 2023-04-13 02:55:27,219 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:55:27,220 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:55:27,222 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4430 sec 2023-04-13 02:55:28,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 02:55:28,888 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-13 02:55:28,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:55:28,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,890 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 02:55:28,891 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,895 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:28,897 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits] 2023-04-13 02:55:28,899 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/233ed51e53104d999ba35c81616cfb94 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/233ed51e53104d999ba35c81616cfb94 2023-04-13 02:55:28,900 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5733bbea979948aaa185c3ae096c7d0d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/5733bbea979948aaa185c3ae096c7d0d 2023-04-13 02:55:28,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a5055e1164924d6483abb0dcd2581e08 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a5055e1164924d6483abb0dcd2581e08 2023-04-13 02:55:28,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a9a158086e754ec19b6b03e25e6e1315 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/A/a9a158086e754ec19b6b03e25e6e1315 2023-04-13 02:55:28,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3e625b11553e4174ae9aa6af680e9827 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/3e625b11553e4174ae9aa6af680e9827 2023-04-13 02:55:28,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/5fc0910a4fbf4da2b3ff454ae180ab3b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/5fc0910a4fbf4da2b3ff454ae180ab3b 2023-04-13 02:55:28,906 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bd01622192054a50a35f9da3d3b5f9bb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/bd01622192054a50a35f9da3d3b5f9bb 2023-04-13 02:55:28,907 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/c45f73218dd842638f59dede408ec14d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/B/c45f73218dd842638f59dede408ec14d 2023-04-13 02:55:28,908 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/24e12734d48647ec8c6cd6786f9e7695 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/24e12734d48647ec8c6cd6786f9e7695 2023-04-13 02:55:28,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/42d89805131a40598af2febbb04166dc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/42d89805131a40598af2febbb04166dc 2023-04-13 02:55:28,910 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9b402c7fd182459bae5454224be25b9d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/9b402c7fd182459bae5454224be25b9d 2023-04-13 02:55:28,911 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f0000277ec00409ca06d1079db66819b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/C/f0000277ec00409ca06d1079db66819b 2023-04-13 02:55:28,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits/533.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a/recovered.edits/533.seqid 2023-04-13 02:55:28,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/6ffe91e2c6ce88591f4f17c4b5e2f40a 2023-04-13 02:55:28,913 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:28,914 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:55:28,914 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 02:55:28,916 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413059172fb37b845cea59146408452f235 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413059172fb37b845cea59146408452f235 2023-04-13 02:55:28,917 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306eac09208344174bbf334728481e3c3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306eac09208344174bbf334728481e3c3 2023-04-13 02:55:28,918 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130774235dc2274f688b140b2d283c891a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130774235dc2274f688b140b2d283c891a 2023-04-13 02:55:28,918 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413190b549972ef4ca7a3f956150da42c9f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413190b549972ef4ca7a3f956150da42c9f 2023-04-13 02:55:28,919 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413219a0948e6b94e039265f5de19cbb0dc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413219a0948e6b94e039265f5de19cbb0dc 2023-04-13 02:55:28,920 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324006cc9693e4532b59e72e91f158f57 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324006cc9693e4532b59e72e91f158f57 2023-04-13 02:55:28,922 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fb4dbc121444e7be9cb486f0db9f8e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fb4dbc121444e7be9cb486f0db9f8e 2023-04-13 02:55:28,923 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413286747edcf0e4e3ea291405afac62ab5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413286747edcf0e4e3ea291405afac62ab5 2023-04-13 02:55:28,924 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413425881324f2d435296d8175136f113b8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413425881324f2d435296d8175136f113b8 2023-04-13 02:55:28,925 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136afb34c344854bec899bd8791810f832 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136afb34c344854bec899bd8791810f832 2023-04-13 02:55:28,926 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bfb371ffdcb4779ad0beef6df4e656f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136bfb371ffdcb4779ad0beef6df4e656f 2023-04-13 02:55:28,927 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385d633e067f7453ebd1db715301e48dd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385d633e067f7453ebd1db715301e48dd 2023-04-13 02:55:28,928 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041390bb87840b104daeb778666304ce4f97 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041390bb87840b104daeb778666304ce4f97 2023-04-13 02:55:28,930 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139c8bc1e0a1584d8a9afcec64daccb319 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304139c8bc1e0a1584d8a9afcec64daccb319 2023-04-13 02:55:28,931 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afdafe144beb45f1b93e907afeed8488 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afdafe144beb45f1b93e907afeed8488 2023-04-13 02:55:28,932 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b9222bb1931d43f58c2da64b8808c1df to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b9222bb1931d43f58c2da64b8808c1df 2023-04-13 02:55:28,933 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df1e0090cbdd4d74b2ddf9183b224a24 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df1e0090cbdd4d74b2ddf9183b224a24 2023-04-13 02:55:28,934 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e534d836c5ca4cfe8678e6c0de6893f4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e534d836c5ca4cfe8678e6c0de6893f4 2023-04-13 02:55:28,935 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ef737cd84a314ef2be57acf7bc675978 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ef737cd84a314ef2be57acf7bc675978 2023-04-13 02:55:28,936 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5ef376ff9754bc8b6e92f36d4061a31 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5ef376ff9754bc8b6e92f36d4061a31 2023-04-13 02:55:28,937 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f8658a25fba54863a855a563ee53eff5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f8658a25fba54863a855a563ee53eff5 2023-04-13 02:55:28,938 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:55:28,939 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,941 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:55:28,942 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:55:28,944 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,944 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:55:28,944 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354528944"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:28,946 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:55:28,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6ffe91e2c6ce88591f4f17c4b5e2f40a, NAME => 'TestAcidGuarantees,,1681354502032.6ffe91e2c6ce88591f4f17c4b5e2f40a.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:55:28,946 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:55:28,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354528946"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:28,948 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:55:28,950 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:28,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-04-13 02:55:28,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 02:55:28,992 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-13 02:55:29,003 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 227), OpenFileDescriptor=562 (was 532) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=398 (was 351) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=783 (was 850) 2023-04-13 02:55:29,014 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=562, MaxFileDescriptor=60000, SystemLoadAverage=398, ProcessCount=170, AvailableMemoryMB=783 2023-04-13 02:55:29,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:55:29,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:55:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:29,018 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:55:29,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-13 02:55:29,019 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:55:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:29,020 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:29,021 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 empty. 2023-04-13 02:55:29,021 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:29,021 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:29,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:29,428 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:55:29,430 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9a7666d97e523c69aa839344392ace09, NAME => 'TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:55:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9a7666d97e523c69aa839344392ace09, disabling compactions & flushes 2023-04-13 02:55:29,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. after waiting 0 ms 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:29,846 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:29,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:29,848 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:55:29,849 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354529849"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354529849"}]},"ts":"1681354529849"} 2023-04-13 02:55:29,850 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:55:29,851 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:55:29,851 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354529851"}]},"ts":"1681354529851"} 2023-04-13 02:55:29,852 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:55:29,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, ASSIGN}] 2023-04-13 02:55:29,857 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, ASSIGN 2023-04-13 02:55:29,857 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:55:30,008 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=9a7666d97e523c69aa839344392ace09, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:30,009 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354530008"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354530008"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354530008"}]},"ts":"1681354530008"} 2023-04-13 02:55:30,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:30,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:30,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:30,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a7666d97e523c69aa839344392ace09, NAME => 'TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:55:30,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:30,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,166 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,167 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A 2023-04-13 02:55:30,167 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A 2023-04-13 02:55:30,167 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:30,167 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a7666d97e523c69aa839344392ace09 columnFamilyName A 2023-04-13 02:55:30,168 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(310): Store=9a7666d97e523c69aa839344392ace09/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:30,168 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,169 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B 2023-04-13 02:55:30,169 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B 2023-04-13 02:55:30,170 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:30,170 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a7666d97e523c69aa839344392ace09 columnFamilyName B 2023-04-13 02:55:30,170 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(310): Store=9a7666d97e523c69aa839344392ace09/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:30,170 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,171 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C 2023-04-13 02:55:30,171 DEBUG [StoreOpener-9a7666d97e523c69aa839344392ace09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C 2023-04-13 02:55:30,172 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:30,172 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a7666d97e523c69aa839344392ace09 columnFamilyName C 2023-04-13 02:55:30,172 INFO [StoreOpener-9a7666d97e523c69aa839344392ace09-1] regionserver.HStore(310): Store=9a7666d97e523c69aa839344392ace09/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:30,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:30,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:55:30,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:30,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:55:30,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a7666d97e523c69aa839344392ace09; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70982480, jitterRate=0.05772137641906738}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:55:30,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:30,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., pid=49, masterSystemTime=1681354530161 2023-04-13 02:55:30,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:30,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:30,179 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=9a7666d97e523c69aa839344392ace09, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:30,180 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354530179"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354530179"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354530179"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354530179"}]},"ts":"1681354530179"} 2023-04-13 02:55:30,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-13 02:55:30,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 in 171 msec 2023-04-13 02:55:30,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-13 02:55:30,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, ASSIGN in 327 msec 2023-04-13 02:55:30,183 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:55:30,184 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354530183"}]},"ts":"1681354530183"} 2023-04-13 02:55:30,184 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:55:30,187 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:55:30,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-13 02:55:31,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 02:55:31,124 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-13 02:55:31,124 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1df28b68 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,129 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fb41c50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,130 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,132 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,133 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:55:31,135 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37604, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:55:31,136 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5cd71f to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,140 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@357cb237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,140 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1f0735dc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,144 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4539ed0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,144 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x14c209bc to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,149 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32cd672d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,149 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x0db388d3 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,152 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b8ff30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,152 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x5de1e3c7 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,156 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c050db8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,156 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x28e3107b to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,159 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cefb012, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,159 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x14f9adf3 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,164 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24a917df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,164 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x21359a0e to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,168 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78fa3dee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,168 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x3bbac0b5 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,174 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36d2d826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,174 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x05a300b1 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:31,178 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6871d7d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:31,184 DEBUG [hconnection-0x77d3b822-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,184 DEBUG [hconnection-0x2f5a6f82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,184 DEBUG [hconnection-0x6463ddef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,184 DEBUG [hconnection-0x6bc363c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,184 DEBUG [hconnection-0x77660401-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,184 DEBUG [hconnection-0x2d3a968a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,185 DEBUG [hconnection-0x3ad66a48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,186 DEBUG [hconnection-0xe264b78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,186 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,187 DEBUG [hconnection-0x3723dbd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,188 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,193 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,193 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,194 DEBUG [hconnection-0xd54f597-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:31,194 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,194 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,196 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,196 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,199 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,199 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:31,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:31,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:55:31,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,213 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:31,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:31,216 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354591224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354591228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354591229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354591232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354591233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,236 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6638c4bb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,240 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,245 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,245 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,245 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,249 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,249 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,249 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,250 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,251 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,255 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,255 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,255 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354591330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354591330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354591333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354591334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354591334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,355 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:31,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:31,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,358 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,360 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,379 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4797e6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,382 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,382 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,383 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,385 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,385 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,385 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,388 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,389 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,391 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:31,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,392 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,392 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:31,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,492 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:31,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:31,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,504 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,505 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,506 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,507 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,507 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,521 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,522 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,532 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,532 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c0b44f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,539 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,542 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,543 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354591538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354591539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354591539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354591539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354591540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,548 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,548 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:31,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/69a3177c770149eaaff2f1d8297ddac0 2023-04-13 02:55:31,649 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:31,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:31,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,656 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,657 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,672 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,672 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,674 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b49843[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,678 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:31,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,693 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,693 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:31,793 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:31,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:31,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,796 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,798 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39099ad2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,812 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,812 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,812 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,817 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,818 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,818 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:31,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354591845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354591846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354591846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354591846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354591846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,918 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:31,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:31,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:31,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:31,922 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:31,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:31,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:31,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:31,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:31,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:31,928 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,928 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:31,928 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:31,928 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:31,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:31,930 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:31,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f67d2b2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:31,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:31,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,933 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:31,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:31,937 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:31,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:31,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:31,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:31,937 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:31,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:31,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,941 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:31,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:31,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:31,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:31,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:31,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:31,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:31,944 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:31,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:31,944 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:32,044 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:32,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:32,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:32,047 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:32,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:32,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:32,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:32,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:32,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:32,050 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,050 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:32,051 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:32,051 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:32,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:32,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c379dd[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:32,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:32,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:32,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:32,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:32,060 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:32,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:32,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:32,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:32,060 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:32,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:32,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:32,065 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,066 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:32,066 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:32,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/953a371b67b9471893c4099399e9cf45 2023-04-13 02:55:32,166 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:32,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:32,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:32,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:32,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:32,169 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:32,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:32,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:32,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:32,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:32,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:32,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:32,173 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:32,174 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,174 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:32,174 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:32,174 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:32,174 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:32,175 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,175 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,176 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:32,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:32,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54089226[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:32,177 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,178 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:32,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:32,178 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:32,180 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,180 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:32,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:32,182 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:32,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:32,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:32,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:32,182 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:32,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,191 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,191 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:32,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,192 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,192 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:32,192 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,292 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:32,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:32,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:32,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:32,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:32,295 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:32,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,296 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:32,296 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:32,296 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,296 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:32,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:32,299 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,299 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:32,299 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:32,299 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:32,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:32,302 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,302 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:32,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21224d7f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:32,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:32,304 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,305 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:32,305 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:32,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:32,305 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:32,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:32,307 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,307 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:32,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:32,310 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:32,310 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:32,310 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:32,310 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:32,310 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:32,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,311 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:32,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:32,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,317 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:32,317 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:32,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354592348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354592348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354592349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:32,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354592349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354592352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,417 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:32,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:32,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:32,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:32,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:32,420 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:32,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:32,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:32,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:32,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:32,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:32,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:32,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:32,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:32,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:32,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,427 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,427 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c1e9ab[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:32,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:32,428 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:32,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:32,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:32,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:32,431 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,431 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:32,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:32,433 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:32,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:32,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:32,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:32,434 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:32,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:32,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,437 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:32,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:32,439 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:32,439 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:32,439 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:32,439 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:32,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:32,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/b79221e1526b4cb5b8d427803d6dfb63 2023-04-13 02:55:32,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/69a3177c770149eaaff2f1d8297ddac0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0 2023-04-13 02:55:32,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0, entries=250, sequenceid=15, filesize=16.2 K 2023-04-13 02:55:32,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/953a371b67b9471893c4099399e9cf45 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45 2023-04-13 02:55:32,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 02:55:32,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/b79221e1526b4cb5b8d427803d6dfb63 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63 2023-04-13 02:55:32,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 02:55:32,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 9a7666d97e523c69aa839344392ace09 in 1290ms, sequenceid=15, compaction requested=false 2023-04-13 02:55:32,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:32,539 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:32,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:32,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:32,542 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:32,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:32,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:32,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:32,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:32,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:32,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:32,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:32,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:32,545 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:32,546 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:32,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:32,746 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:55:32,954 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/8fcf1a7a9e4b4ee58c47fec262b56a84 2023-04-13 02:55:32,965 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/f6b61b60482c4428bec4a7ff94ded037 2023-04-13 02:55:33,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:33,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:33,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354593364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354593365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354593366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354593367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354593368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,374 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/9524f111029a4026ba9ec4bf26a70ed5 2023-04-13 02:55:33,377 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/8fcf1a7a9e4b4ee58c47fec262b56a84 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84 2023-04-13 02:55:33,381 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84, entries=150, sequenceid=34, filesize=11.6 K 2023-04-13 02:55:33,382 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/f6b61b60482c4428bec4a7ff94ded037 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037 2023-04-13 02:55:33,390 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037, entries=150, sequenceid=34, filesize=11.6 K 2023-04-13 02:55:33,390 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/9524f111029a4026ba9ec4bf26a70ed5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5 2023-04-13 02:55:33,396 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5, entries=150, sequenceid=34, filesize=11.6 K 2023-04-13 02:55:33,396 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9a7666d97e523c69aa839344392ace09 in 850ms, sequenceid=34, compaction requested=false 2023-04-13 02:55:33,396 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:33,397 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:33,397 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:33,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:33,400 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:33,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423e63ff[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:33,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:33,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,404 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:33,404 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:33,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:33,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:33,406 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:33,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:33,409 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:33,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:33,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:33,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:33,409 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:33,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,419 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:33,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:33,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,427 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:33,427 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:33,427 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,427 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,427 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,427 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,474 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:33,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:33,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6 2023-04-13 02:55:33,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354593500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354593505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354593507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354593508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354593509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,527 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:33,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:33,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:33,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:33,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:33,530 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:33,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:33,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:33,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,533 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:33,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:33,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:33,534 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,534 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:33,534 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:33,534 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:33,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:33,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,537 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:33,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b7503c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:33,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:33,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:33,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:33,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:33,540 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:33,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:33,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,543 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:33,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:33,546 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:33,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:33,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:33,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:33,546 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:33,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:33,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:33,555 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:33,555 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:33,555 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354593610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354593610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354593614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354593614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354593615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,655 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:33,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:33,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:33,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:33,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:33,658 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:33,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:33,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:33,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:33,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:33,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:33,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:33,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:33,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:33,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:33,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:33,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:33,665 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:33,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:33,669 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,669 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,670 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:33,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@614da6a0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:33,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:33,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:33,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:33,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:33,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:33,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:33,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:33,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:33,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:33,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:33,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:33,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:33,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:33,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:33,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,697 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:33,697 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:33,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,797 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:33,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:33,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:33,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:33,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:33,800 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:33,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:33,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:33,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,802 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:33,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:33,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:33,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:33,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:33,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:33,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:33,807 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7e8793[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:33,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:33,809 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:33,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:33,812 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:33,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:33,815 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:33,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:33,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:33,815 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:33,815 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:33,817 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,817 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,817 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354593814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354593814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354593820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:33,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:33,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,823 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,824 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:33,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:33,824 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:33,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,824 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354593820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354593821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3a30d8363bd34bdaa9bd5bd0395fef69 2023-04-13 02:55:33,924 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:33,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:33,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:33,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:33,929 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:33,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:33,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:33,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e8efba4858b94450852b0e0ac99e20fc 2023-04-13 02:55:33,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:33,936 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:33,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:33,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:33,938 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:33,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6 2023-04-13 02:55:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6, entries=150, sequenceid=48, filesize=11.6 K 2023-04-13 02:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3a30d8363bd34bdaa9bd5bd0395fef69 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69 2023-04-13 02:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fee3638[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:33,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:33,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:33,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:33,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:33,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:33,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69, entries=100, sequenceid=48, filesize=9.3 K 2023-04-13 02:55:33,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:33,953 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:33,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e8efba4858b94450852b0e0ac99e20fc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc 2023-04-13 02:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:33,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:33,958 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:33,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:33,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:33,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:33,958 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:33,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:33,964 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:33,964 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:33,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:33,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:33,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:33,964 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:33,964 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:33,964 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:33,964 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:33,964 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:33,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:33,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:33,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:33,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:33,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc, entries=100, sequenceid=48, filesize=9.3 K 2023-04-13 02:55:33,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9a7666d97e523c69aa839344392ace09 in 494ms, sequenceid=48, compaction requested=true 2023-04-13 02:55:33,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:33,968 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:33,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:33,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:33,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:33,969 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:33,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:33,970 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:33,970 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40334 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:33,970 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:33,970 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:33,970 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,970 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:33,970 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=32.5 K 2023-04-13 02:55:33,970 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=39.4 K 2023-04-13 02:55:33,970 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 953a371b67b9471893c4099399e9cf45, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681354531208 2023-04-13 02:55:33,971 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 69a3177c770149eaaff2f1d8297ddac0, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681354531208 2023-04-13 02:55:33,971 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f6b61b60482c4428bec4a7ff94ded037, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681354531223 2023-04-13 02:55:33,971 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3a30d8363bd34bdaa9bd5bd0395fef69, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354533366 2023-04-13 02:55:33,971 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8fcf1a7a9e4b4ee58c47fec262b56a84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681354531223 2023-04-13 02:55:33,972 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ddfd5e1f13c24f7b8312f4c3dacfd3f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354533366 2023-04-13 02:55:33,993 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#366 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:33,995 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#367 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:34,039 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/41ea4fac9dba4eefa12c6406251dde3d] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:55:34,046 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1f4b0793425241aba5e3726eca3795c5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1f4b0793425241aba5e3726eca3795c5 2023-04-13 02:55:34,047 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/41ea4fac9dba4eefa12c6406251dde3d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/41ea4fac9dba4eefa12c6406251dde3d 2023-04-13 02:55:34,051 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 1f4b0793425241aba5e3726eca3795c5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:34,051 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:34,051 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354533969; duration=0sec 2023-04-13 02:55:34,051 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:34,051 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:34,052 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into 41ea4fac9dba4eefa12c6406251dde3d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:34,052 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:34,052 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=13, startTime=1681354533968; duration=0sec 2023-04-13 02:55:34,053 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:34,053 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:34,053 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:34,053 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,053 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=32.5 K 2023-04-13 02:55:34,054 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b79221e1526b4cb5b8d427803d6dfb63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681354531208 2023-04-13 02:55:34,054 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9524f111029a4026ba9ec4bf26a70ed5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681354531223 2023-04-13 02:55:34,054 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting e8efba4858b94450852b0e0ac99e20fc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354533366 2023-04-13 02:55:34,060 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#368 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:34,064 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,069 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/cec8bde4e6824d6ea64e902753c9850a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cec8bde4e6824d6ea64e902753c9850a 2023-04-13 02:55:34,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,071 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,074 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into cec8bde4e6824d6ea64e902753c9850a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:34,074 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:34,074 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=13, startTime=1681354533969; duration=0sec 2023-04-13 02:55:34,074 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:34,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,079 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,080 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:34,080 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,097 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d 2023-04-13 02:55:34,118 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4f6cb1a262ab42578bad4e7c75863996 2023-04-13 02:55:34,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:34,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,130 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/98c6128f624d473994d63938b894ddc2 2023-04-13 02:55:34,133 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d 2023-04-13 02:55:34,137 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d, entries=150, sequenceid=69, filesize=11.6 K 2023-04-13 02:55:34,142 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4f6cb1a262ab42578bad4e7c75863996 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996 2023-04-13 02:55:34,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354594135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354594136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354594139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,145 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996, entries=150, sequenceid=69, filesize=11.6 K 2023-04-13 02:55:34,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/98c6128f624d473994d63938b894ddc2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2 2023-04-13 02:55:34,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354594142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354594144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,149 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2, entries=150, sequenceid=69, filesize=11.6 K 2023-04-13 02:55:34,150 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 9a7666d97e523c69aa839344392ace09 in 71ms, sequenceid=69, compaction requested=false 2023-04-13 02:55:34,150 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:34,150 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:34,152 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730ad49[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:34,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:34,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,155 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:34,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:34,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:34,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:34,157 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:34,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:34,160 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:34,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:34,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:34,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:34,160 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:34,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,165 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:34,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:34,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:34,168 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:34,168 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:34,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,168 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,169 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:34,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:55:34,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:34,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:34,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:34,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/4febc13a5fdb4868a2c09eae44999fab 2023-04-13 02:55:34,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354594260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354594261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354594262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,268 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354594263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354594264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,284 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,288 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,288 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,289 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,291 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:34,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:34,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,294 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:34,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc473a9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:34,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:34,295 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:34,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:34,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:34,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:34,299 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,299 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:34,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:34,301 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:34,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:34,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:34,302 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:34,302 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:34,304 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,305 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,305 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,306 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:34,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:34,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,308 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,309 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:34,309 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:34,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354594365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354594369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354594369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354594370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354594372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,409 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,413 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,416 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,417 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,418 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,419 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:34,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212c3f5d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:34,425 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:34,425 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,426 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:34,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:34,426 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:34,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:34,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,429 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:34,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:34,432 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:34,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:34,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:34,432 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:34,432 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:34,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,433 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,437 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:34,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:34,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,438 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,438 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:34,538 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,541 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,544 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:34,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:34,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,547 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,548 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:34,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fed92b6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:34,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:34,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,550 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:34,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:34,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:34,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,551 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:34,552 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:34,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:34,557 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:34,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:34,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:34,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:34,557 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:34,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,562 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,568 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:34,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:34,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,569 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:34,569 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:34,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:34,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354594574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354594575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354594577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354594577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354594582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,669 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/cdf85e494d9b4f2e987a09d4bfa95934 2023-04-13 02:55:34,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,673 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,676 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,677 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,678 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,678 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:34,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d353ae7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:34,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:34,681 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:34,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:34,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,688 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:34,688 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,688 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/f7bad02709124e169a5e351c52d1df30 2023-04-13 02:55:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:34,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:34,692 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:34,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:34,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:34,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:34,692 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:34,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/4febc13a5fdb4868a2c09eae44999fab as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab 2023-04-13 02:55:34,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,695 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:34,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab, entries=250, sequenceid=86, filesize=16.2 K 2023-04-13 02:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/cdf85e494d9b4f2e987a09d4bfa95934 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:34,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:34,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:34,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,701 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:34,701 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:34,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:34,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:34,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934, entries=150, sequenceid=86, filesize=11.6 K 2023-04-13 02:55:34,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/f7bad02709124e169a5e351c52d1df30 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30 2023-04-13 02:55:34,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30, entries=150, sequenceid=86, filesize=11.6 K 2023-04-13 02:55:34,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a7666d97e523c69aa839344392ace09 in 462ms, sequenceid=86, compaction requested=true 2023-04-13 02:55:34,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:34,711 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:34,711 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:34,712 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:34,712 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:34,712 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,712 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/41ea4fac9dba4eefa12c6406251dde3d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=39.5 K 2023-04-13 02:55:34,712 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:34,712 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:34,712 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,713 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1f4b0793425241aba5e3726eca3795c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=34.9 K 2023-04-13 02:55:34,713 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 41ea4fac9dba4eefa12c6406251dde3d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354531223 2023-04-13 02:55:34,713 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1f4b0793425241aba5e3726eca3795c5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354531223 2023-04-13 02:55:34,713 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9ea9ef7e5fe64afc9b16f7f3d2e1e78d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681354533498 2023-04-13 02:55:34,713 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4f6cb1a262ab42578bad4e7c75863996, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681354533498 2023-04-13 02:55:34,713 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4febc13a5fdb4868a2c09eae44999fab, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:34,714 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cdf85e494d9b4f2e987a09d4bfa95934, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:34,720 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#375 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:34,723 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#376 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:34,801 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:34,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:34,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:34,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:34,805 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:34,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:34,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:34,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:34,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:34,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:34,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:34,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:34,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:34,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:34,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:34,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:34,808 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:34,809 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:34,809 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:34,816 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/dc3c78af216c46f9b124fd9045fc26af 2023-04-13 02:55:34,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:34,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:34,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354594897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354594903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354594904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354594905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:34,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354594905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354595006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354595012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354595014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354595018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354595018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,141 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1201e5e589644195968b5f7828f06374 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1201e5e589644195968b5f7828f06374 2023-04-13 02:55:35,159 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/abca624643a043d780dd8a94b8a83c67 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/abca624643a043d780dd8a94b8a83c67 2023-04-13 02:55:35,164 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 1201e5e589644195968b5f7828f06374(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:35,164 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:35,164 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354534711; duration=0sec 2023-04-13 02:55:35,164 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:35,164 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:35,167 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:35,167 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:35,167 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:35,167 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cec8bde4e6824d6ea64e902753c9850a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=34.9 K 2023-04-13 02:55:35,167 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cec8bde4e6824d6ea64e902753c9850a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681354531223 2023-04-13 02:55:35,168 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into abca624643a043d780dd8a94b8a83c67(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:35,168 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:35,168 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=13, startTime=1681354534711; duration=0sec 2023-04-13 02:55:35,168 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 98c6128f624d473994d63938b894ddc2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681354533498 2023-04-13 02:55:35,168 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:35,172 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f7bad02709124e169a5e351c52d1df30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:35,182 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#379 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 02:55:35,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354595216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354595220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354595220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,227 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/be96534590be43bab7e1bdb07d37d66f 2023-04-13 02:55:35,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354595226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354595226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354595524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354595527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354595529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354595533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:35,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354595535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,603 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/715f1c30f8564a7ab5230d254d8d5de0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/715f1c30f8564a7ab5230d254d8d5de0 2023-04-13 02:55:35,606 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into 715f1c30f8564a7ab5230d254d8d5de0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:35,606 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:35,607 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=13, startTime=1681354534711; duration=0sec 2023-04-13 02:55:35,607 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:35,648 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc 2023-04-13 02:55:35,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/dc3c78af216c46f9b124fd9045fc26af as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af 2023-04-13 02:55:35,656 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af, entries=150, sequenceid=101, filesize=11.6 K 2023-04-13 02:55:35,659 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/be96534590be43bab7e1bdb07d37d66f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f 2023-04-13 02:55:35,662 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f, entries=150, sequenceid=101, filesize=11.6 K 2023-04-13 02:55:35,663 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc 2023-04-13 02:55:35,668 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc, entries=150, sequenceid=101, filesize=11.6 K 2023-04-13 02:55:35,669 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 860ms, sequenceid=101, compaction requested=false 2023-04-13 02:55:35,669 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:35,669 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:35,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:35,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:35,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c45fd3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:35,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:35,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:35,677 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:35,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:35,682 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:35,684 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:35,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:35,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:35,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:35,685 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:35,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:35,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:35,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:35,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:35,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:35,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:35,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:35,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:35,690 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:35,690 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:35,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:35,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:35,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,791 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:35,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:35,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:35,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:35,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:35,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:35,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:35,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:35,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:35,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:35,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:35,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:35,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:35,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:35,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:35,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:35,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:35,799 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:35,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:35,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:35,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:35,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:35,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:35,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:36,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:36,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354596048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354596049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354596050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354596050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354596051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354596156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354596156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354596157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354596157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354596157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,212 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/800bcdb120c84a6997fdc757feda45c5 2023-04-13 02:55:36,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354596359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354596361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354596363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354596363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354596363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,627 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/de8a3af6d5cc42e2882463c701f30bd7 2023-04-13 02:55:36,642 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/1644f119266249f485a3adf1e46fb05f 2023-04-13 02:55:36,649 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/800bcdb120c84a6997fdc757feda45c5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5 2023-04-13 02:55:36,653 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5, entries=150, sequenceid=121, filesize=11.6 K 2023-04-13 02:55:36,654 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/de8a3af6d5cc42e2882463c701f30bd7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7 2023-04-13 02:55:36,661 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7, entries=150, sequenceid=121, filesize=11.6 K 2023-04-13 02:55:36,662 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/1644f119266249f485a3adf1e46fb05f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f 2023-04-13 02:55:36,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354596666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354596667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354596669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354596671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354596671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,677 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f, entries=150, sequenceid=121, filesize=11.6 K 2023-04-13 02:55:36,677 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a7666d97e523c69aa839344392ace09 in 878ms, sequenceid=121, compaction requested=true 2023-04-13 02:55:36,678 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:36,678 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:36,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:36,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:36,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:36,678 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:36,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:36,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:36,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55524ce1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:36,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:36,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,683 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:36,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:36,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:36,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:36,688 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:36,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:36,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:36,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:36,688 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:36,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:36,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:36,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:36,697 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:36,697 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:36,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:36,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:36,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:36,797 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:36,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:36,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:36,800 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:36,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:36,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:36,800 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:36,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:36,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,802 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:36,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:36,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:36,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:36,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:36,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:36,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:36,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:36,803 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:36,804 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:36,804 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:36,805 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:36,805 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:36,805 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:37,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:37,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354597190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354597190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354597195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354597195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354597195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,217 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f0d7746bfa6f4d1c8c4113a95c9d3283 2023-04-13 02:55:37,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354597298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354597298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354597302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354597302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354597302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354597508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354597508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354597512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354597513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354597513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,632 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/f3087dfbd0874bf3ad31ddef77971d93 2023-04-13 02:55:37,642 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/bf106c628d9447c1acbabf27d587b9b4 2023-04-13 02:55:37,646 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f0d7746bfa6f4d1c8c4113a95c9d3283 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283 2023-04-13 02:55:37,649 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283, entries=150, sequenceid=136, filesize=11.8 K 2023-04-13 02:55:37,650 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/f3087dfbd0874bf3ad31ddef77971d93 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93 2023-04-13 02:55:37,653 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93, entries=150, sequenceid=136, filesize=11.8 K 2023-04-13 02:55:37,653 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/bf106c628d9447c1acbabf27d587b9b4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4 2023-04-13 02:55:37,657 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4, entries=150, sequenceid=136, filesize=11.8 K 2023-04-13 02:55:37,658 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 853ms, sequenceid=136, compaction requested=true 2023-04-13 02:55:37,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:37,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:37,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:37,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:37,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:37,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:37,659 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:37,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:37,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:37,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f90cdf[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:37,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:37,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:37,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:37,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:37,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:37,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:37,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:37,666 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:37,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:37,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:37,666 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:37,667 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:37,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:37,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:37,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:37,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:37,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:37,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:37,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:37,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:37,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:37,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:37,672 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:37,672 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:37,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:37,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:37,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:37,772 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:37,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:37,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:37,775 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:37,776 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:37,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:37,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:37,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:37,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:37,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:37,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:37,779 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:37,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:37,787 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/1cbce6fb72734b83be90d9f2dd3a8fff 2023-04-13 02:55:37,796 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/b5bf1e38031d4359b07dd7b5920aaf0a 2023-04-13 02:55:37,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:37,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:37,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354597831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354597831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354597832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354597833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354597834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/41ea4fac9dba4eefa12c6406251dde3d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab] to archive 2023-04-13 02:55:37,857 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:37,859 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/69a3177c770149eaaff2f1d8297ddac0 2023-04-13 02:55:37,860 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8fcf1a7a9e4b4ee58c47fec262b56a84 2023-04-13 02:55:37,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/41ea4fac9dba4eefa12c6406251dde3d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/41ea4fac9dba4eefa12c6406251dde3d 2023-04-13 02:55:37,862 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ddfd5e1f13c24f7b8312f4c3dacfd3f6 2023-04-13 02:55:37,863 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9ea9ef7e5fe64afc9b16f7f3d2e1e78d 2023-04-13 02:55:37,864 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4febc13a5fdb4868a2c09eae44999fab 2023-04-13 02:55:37,866 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1f4b0793425241aba5e3726eca3795c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934] to archive 2023-04-13 02:55:37,867 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:37,868 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/953a371b67b9471893c4099399e9cf45 2023-04-13 02:55:37,869 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f6b61b60482c4428bec4a7ff94ded037 2023-04-13 02:55:37,870 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1f4b0793425241aba5e3726eca3795c5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1f4b0793425241aba5e3726eca3795c5 2023-04-13 02:55:37,871 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3a30d8363bd34bdaa9bd5bd0395fef69 2023-04-13 02:55:37,873 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4f6cb1a262ab42578bad4e7c75863996 2023-04-13 02:55:37,874 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cdf85e494d9b4f2e987a09d4bfa95934 2023-04-13 02:55:37,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cec8bde4e6824d6ea64e902753c9850a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30] to archive 2023-04-13 02:55:37,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:37,878 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/b79221e1526b4cb5b8d427803d6dfb63 2023-04-13 02:55:37,879 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/9524f111029a4026ba9ec4bf26a70ed5 2023-04-13 02:55:37,880 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cec8bde4e6824d6ea64e902753c9850a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cec8bde4e6824d6ea64e902753c9850a 2023-04-13 02:55:37,881 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e8efba4858b94450852b0e0ac99e20fc 2023-04-13 02:55:37,882 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/98c6128f624d473994d63938b894ddc2 2023-04-13 02:55:37,883 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f7bad02709124e169a5e351c52d1df30 2023-04-13 02:55:37,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354597935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354597941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354597941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354597941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:37,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:37,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354597941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354598144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354598146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354598147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354598147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354598151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,206 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/7b22cc422958487cb734cfcd5b686555 2023-04-13 02:55:38,211 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/1cbce6fb72734b83be90d9f2dd3a8fff as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff 2023-04-13 02:55:38,215 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff, entries=150, sequenceid=153, filesize=11.8 K 2023-04-13 02:55:38,216 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/b5bf1e38031d4359b07dd7b5920aaf0a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a 2023-04-13 02:55:38,219 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a, entries=150, sequenceid=153, filesize=11.8 K 2023-04-13 02:55:38,220 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/7b22cc422958487cb734cfcd5b686555 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555 2023-04-13 02:55:38,223 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555, entries=150, sequenceid=153, filesize=11.8 K 2023-04-13 02:55:38,223 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a7666d97e523c69aa839344392ace09 in 444ms, sequenceid=153, compaction requested=true 2023-04-13 02:55:38,224 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:38,224 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:38,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:38,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,227 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:38,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128fece5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:38,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:38,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,230 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:38,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:38,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:38,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:38,234 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:38,234 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:38,234 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:38,234 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:38,234 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:38,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:38,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:38,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:38,246 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:38,246 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:38,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:38,246 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:38,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:38,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:38,346 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:38,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:38,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:38,350 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:38,350 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:38,350 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:38,350 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:38,351 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,351 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,352 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,352 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:38,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:38,353 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,354 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:38,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:38,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:38,355 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:38,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:38,356 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:38,356 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:38,356 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,374 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0d9362e2fcce4b60bde5d541c8c21121 2023-04-13 02:55:38,394 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/9b16d1e59f26437aa0191f56271aadbb 2023-04-13 02:55:38,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:38,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:38,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354598468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354598469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354598470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354598476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354598482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354598584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354598584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354598584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354598584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354598591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354598796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354598796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354598796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354598797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:38,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354598798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,822 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/ed24aa44b657454fb8bff49f5c50753e 2023-04-13 02:55:38,826 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0d9362e2fcce4b60bde5d541c8c21121 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121 2023-04-13 02:55:38,830 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121, entries=150, sequenceid=168, filesize=11.8 K 2023-04-13 02:55:38,830 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/9b16d1e59f26437aa0191f56271aadbb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb 2023-04-13 02:55:38,834 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb, entries=150, sequenceid=168, filesize=11.8 K 2023-04-13 02:55:38,835 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/ed24aa44b657454fb8bff49f5c50753e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e 2023-04-13 02:55:38,839 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e, entries=150, sequenceid=168, filesize=11.8 K 2023-04-13 02:55:38,840 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 484ms, sequenceid=168, compaction requested=true 2023-04-13 02:55:38,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:38,840 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:38,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:38,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:38,840 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:38,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,842 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:38,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@250203a0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:38,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:38,844 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,845 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:38,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,845 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:38,847 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:38,849 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:38,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:38,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:38,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:38,849 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:38,851 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,851 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:38,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:38,852 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:38,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:38,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,856 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:38,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:38,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:38,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:38,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:38,857 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:38,857 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:38,957 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:38,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:38,960 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:38,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:38,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:38,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:38,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,961 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:38,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:38,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:38,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:38,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:38,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:38,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:38,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:38,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:38,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:38,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:38,964 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:38,965 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:38,965 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:38,976 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0bd99a0e75064d0c8721149b0727141e 2023-04-13 02:55:38,985 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/640f6c0ff4f241f58dd118a12a59deb8 2023-04-13 02:55:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:39,110 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:39,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354599125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354599126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354599127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354599133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354599134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354599235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354599235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354599236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354599236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354599249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,394 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/a3765f91ab5244b9a5e1582077b29f00 2023-04-13 02:55:39,399 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0bd99a0e75064d0c8721149b0727141e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e 2023-04-13 02:55:39,403 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e, entries=150, sequenceid=185, filesize=11.8 K 2023-04-13 02:55:39,405 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/640f6c0ff4f241f58dd118a12a59deb8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8 2023-04-13 02:55:39,410 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8, entries=150, sequenceid=185, filesize=11.8 K 2023-04-13 02:55:39,410 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/a3765f91ab5244b9a5e1582077b29f00 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00 2023-04-13 02:55:39,414 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00, entries=150, sequenceid=185, filesize=11.8 K 2023-04-13 02:55:39,415 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 450ms, sequenceid=185, compaction requested=true 2023-04-13 02:55:39,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:39,415 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:39,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:39,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:39,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:39,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,417 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,418 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627076a5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:39,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:39,420 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:39,422 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:39,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:39,423 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,423 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:39,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:39,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:39,426 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:39,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:39,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:39,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:39,426 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:39,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,430 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,441 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:39,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:39,442 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:39,442 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:39,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:39,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:39,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:39,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354599472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354599482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354599482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354599484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354599485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,542 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:39,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:39,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:39,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:39,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:39,545 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:39,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:39,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:39,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,546 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:39,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:39,552 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,552 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:39,552 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:39,553 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:39,555 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2111c0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:39,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:39,557 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,558 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:39,558 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:39,558 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:39,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,558 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:39,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:39,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:39,565 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:39,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:39,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:39,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:39,565 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:39,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,568 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,569 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,570 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,577 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,577 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:39,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:39,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,579 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,579 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,579 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,579 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:39,579 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:39,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354599585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354599598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354599598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354599601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354599601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,679 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:39,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:39,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:39,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:39,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:39,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:39,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:39,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:39,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,684 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:39,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:39,686 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:39,686 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,687 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:39,687 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:39,687 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:39,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:39,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:39,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:39,688 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41242865[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:39,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:39,690 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:39,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:39,702 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:39,702 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:39,702 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:39,702 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:39,702 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:39,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,704 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,704 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,706 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,709 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,710 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,710 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,710 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:39,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,710 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,710 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,710 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:39,710 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:39,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:39,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354599796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,810 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:39,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:39,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354599805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354599806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354599806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:39,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354599807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:39,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:39,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:39,817 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:39,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:39,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:39,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,819 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:39,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:39,824 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,824 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:39,824 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:39,824 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:39,824 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:39,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740e3ce2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:39,828 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:39,828 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,829 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:39,830 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:39,830 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,830 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:39,831 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,831 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:39,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:39,833 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:39,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:39,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:39,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:39,833 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:39,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,837 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,842 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,842 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:39,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:39,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,843 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:39,843 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:39,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/a7446702f2b74c03aa1942f1f865aac2 2023-04-13 02:55:39,943 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:39,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:39,949 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:39,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:39,950 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:39,950 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,950 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:39,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:39,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:39,953 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,953 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:39,953 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:39,953 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:39,955 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152aa998[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:39,957 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:39,957 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:39,959 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:39,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:39,962 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:39,965 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:39,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:39,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:39,965 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:39,965 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:39,967 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,967 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:39,967 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,967 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,968 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:39,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,972 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:39,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:39,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:39,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:39,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:39,972 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:39,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:39,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,973 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:39,973 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:39,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:39,973 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:39,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:39,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:40,073 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:40,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:40,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:40,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:40,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:40,077 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:40,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:40,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:40,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:40,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:40,082 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,082 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:40,088 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,088 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:40,089 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:40,089 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:40,089 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:40,092 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,092 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,092 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,093 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:40,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312105ec[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:40,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:40,094 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:40,095 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,095 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:40,097 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,097 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:40,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:40,100 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:40,100 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:40,100 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:40,100 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:40,100 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:40,103 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,103 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,103 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,105 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:40,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:40,109 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:40,109 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,109 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:40,110 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:40,110 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:40,109 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,110 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:40,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354600104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354600115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354600116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354600117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354600118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,210 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:40,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:40,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:40,213 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:40,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:40,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:40,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,214 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,216 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:40,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:40,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:40,220 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,221 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,221 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:40,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:40,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:40,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:40,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,224 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:40,225 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:40,225 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:40,225 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,225 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:40,226 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:40,226 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:40,226 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:40,227 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,227 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,228 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,228 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:40,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512664ad[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:40,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:40,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,231 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:40,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:40,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:40,231 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:40,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:40,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:40,233 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,233 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:40,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:40,236 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:40,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:40,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:40,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:40,237 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:40,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:40,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:40,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:40,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,240 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:40,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:40,242 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:40,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:40,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:40,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:40,243 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:40,243 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:40,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:40,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/eafc2f2f825b401d81740e0a77ac37bc 2023-04-13 02:55:40,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/216d8aa2efbe4a6f80922375dbaaaeb2 2023-04-13 02:55:40,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/a7446702f2b74c03aa1942f1f865aac2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2 2023-04-13 02:55:40,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2, entries=250, sequenceid=203, filesize=16.4 K 2023-04-13 02:55:40,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/eafc2f2f825b401d81740e0a77ac37bc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc 2023-04-13 02:55:40,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc, entries=150, sequenceid=203, filesize=11.8 K 2023-04-13 02:55:40,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/216d8aa2efbe4a6f80922375dbaaaeb2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2 2023-04-13 02:55:40,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2, entries=150, sequenceid=203, filesize=11.8 K 2023-04-13 02:55:40,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 9a7666d97e523c69aa839344392ace09 in 866ms, sequenceid=203, compaction requested=true 2023-04-13 02:55:40,315 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:40,315 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:40,315 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-13 02:55:40,315 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:40,315 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-13 02:55:40,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:40,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96044 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-13 02:55:40,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:40,319 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,319 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1201e5e589644195968b5f7828f06374, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=93.8 K 2023-04-13 02:55:40,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1201e5e589644195968b5f7828f06374, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:40,320 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting be96534590be43bab7e1bdb07d37d66f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681354534256 2023-04-13 02:55:40,320 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 100824 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-13 02:55:40,320 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:40,320 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,320 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/abca624643a043d780dd8a94b8a83c67, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=98.5 K 2023-04-13 02:55:40,320 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting de8a3af6d5cc42e2882463c701f30bd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681354534902 2023-04-13 02:55:40,320 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting abca624643a043d780dd8a94b8a83c67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:40,321 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f3087dfbd0874bf3ad31ddef77971d93, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681354536049 2023-04-13 02:55:40,321 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dc3c78af216c46f9b124fd9045fc26af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681354534256 2023-04-13 02:55:40,321 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b5bf1e38031d4359b07dd7b5920aaf0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681354537193 2023-04-13 02:55:40,321 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 800bcdb120c84a6997fdc757feda45c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681354534902 2023-04-13 02:55:40,321 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9b16d1e59f26437aa0191f56271aadbb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681354537831 2023-04-13 02:55:40,322 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f0d7746bfa6f4d1c8c4113a95c9d3283, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681354536049 2023-04-13 02:55:40,322 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 640f6c0ff4f241f58dd118a12a59deb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681354538466 2023-04-13 02:55:40,322 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 1cbce6fb72734b83be90d9f2dd3a8fff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681354537193 2023-04-13 02:55:40,322 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting eafc2f2f825b401d81740e0a77ac37bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:40,322 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0d9362e2fcce4b60bde5d541c8c21121, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681354537831 2023-04-13 02:55:40,323 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0bd99a0e75064d0c8721149b0727141e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681354538466 2023-04-13 02:55:40,323 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a7446702f2b74c03aa1942f1f865aac2, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:40,335 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#399 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:40,340 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#400 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:40,348 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:40,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:40,362 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:40,363 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,363 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:40,363 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:40,363 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,363 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,364 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:40,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:40,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:40,364 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,365 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:40,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:40,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:40,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:40,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:40,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:40,366 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:40,367 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/a5150cee686f4b20ab63ad06f4d1b352 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5150cee686f4b20ab63ad06f4d1b352 2023-04-13 02:55:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:40,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:40,368 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:40,368 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:40,375 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into a5150cee686f4b20ab63ad06f4d1b352(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:40,376 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:40,376 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=8, startTime=1681354540315; duration=0sec 2023-04-13 02:55:40,376 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:40,376 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-13 02:55:40,379 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96044 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-13 02:55:40,379 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:40,379 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:40,379 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/715f1c30f8564a7ab5230d254d8d5de0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=93.8 K 2023-04-13 02:55:40,379 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 715f1c30f8564a7ab5230d254d8d5de0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681354534133 2023-04-13 02:55:40,379 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e9d5fb4d7f4d474c8ffa4175e6d2a6bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681354534256 2023-04-13 02:55:40,380 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1644f119266249f485a3adf1e46fb05f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1681354534902 2023-04-13 02:55:40,380 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting bf106c628d9447c1acbabf27d587b9b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681354536049 2023-04-13 02:55:40,381 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7b22cc422958487cb734cfcd5b686555, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681354537193 2023-04-13 02:55:40,381 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ed24aa44b657454fb8bff49f5c50753e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681354537831 2023-04-13 02:55:40,381 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a3765f91ab5244b9a5e1582077b29f00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681354538466 2023-04-13 02:55:40,382 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 216d8aa2efbe4a6f80922375dbaaaeb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:40,397 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:40,408 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/4caf43e521ee4c41a29bb83d83ef5f93 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4caf43e521ee4c41a29bb83d83ef5f93 2023-04-13 02:55:40,413 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into 4caf43e521ee4c41a29bb83d83ef5f93(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:40,413 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:40,413 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=8, startTime=1681354540315; duration=0sec 2023-04-13 02:55:40,413 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:40,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:40,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:40,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354600640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354600645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354600646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354600647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354600648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,750 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/becaaa52da194ac19be960f532653edc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/becaaa52da194ac19be960f532653edc 2023-04-13 02:55:40,756 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into becaaa52da194ac19be960f532653edc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:40,756 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:40,756 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=8, startTime=1681354540315; duration=0sec 2023-04-13 02:55:40,756 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:40,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354600749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354600752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354600757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354600757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354600758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,787 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ac4da4288fbc4b71b9ddc9a0fbae3655 2023-04-13 02:55:40,819 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/0aeffdc7ca8043898e7de9e17bb6881f 2023-04-13 02:55:40,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354600966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354600966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354600967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354600967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:40,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354600967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41040 deadline: 1681354601270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354601275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354601276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40982 deadline: 1681354601277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41032 deadline: 1681354601277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,282 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/f419362c156241e495ed009e573cd9e1 2023-04-13 02:55:41,287 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ac4da4288fbc4b71b9ddc9a0fbae3655 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655 2023-04-13 02:55:41,291 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655, entries=150, sequenceid=218, filesize=11.8 K 2023-04-13 02:55:41,292 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/0aeffdc7ca8043898e7de9e17bb6881f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f 2023-04-13 02:55:41,298 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f, entries=150, sequenceid=218, filesize=11.8 K 2023-04-13 02:55:41,298 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/f419362c156241e495ed009e573cd9e1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1 2023-04-13 02:55:41,305 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1, entries=150, sequenceid=218, filesize=11.8 K 2023-04-13 02:55:41,312 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 944ms, sequenceid=218, compaction requested=false 2023-04-13 02:55:41,312 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:41,312 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:41,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:41,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:41,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:41,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,314 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:41,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54caab47[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:41,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:41,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,318 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:41,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:41,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,319 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:41,320 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,320 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:41,320 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,320 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:41,323 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:41,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:41,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:41,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:41,323 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:41,330 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,330 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,330 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,331 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,333 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,336 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:41,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,336 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:41,336 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:41,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,436 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:41,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:41,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:41,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:41,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:41,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:41,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:41,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:41,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,454 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:41,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:41,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:41,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:41,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:41,484 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:41,484 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,484 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:41,484 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:41,484 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:41,484 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,485 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:41,485 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,485 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:41,485 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,519 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/df44d2bea8484f8f928cfd7c3fc14981 2023-04-13 02:55:41,557 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/9f2e639da5c74faa8835eb7fa829b47e 2023-04-13 02:55:41,604 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/19da798adfa54613b2ab894421e618b1 2023-04-13 02:55:41,611 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/df44d2bea8484f8f928cfd7c3fc14981 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981 2023-04-13 02:55:41,616 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981, entries=150, sequenceid=238, filesize=11.8 K 2023-04-13 02:55:41,617 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/9f2e639da5c74faa8835eb7fa829b47e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e 2023-04-13 02:55:41,624 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e, entries=150, sequenceid=238, filesize=11.8 K 2023-04-13 02:55:41,624 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/19da798adfa54613b2ab894421e618b1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1 2023-04-13 02:55:41,629 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1, entries=150, sequenceid=238, filesize=11.8 K 2023-04-13 02:55:41,630 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 9a7666d97e523c69aa839344392ace09 in 146ms, sequenceid=238, compaction requested=true 2023-04-13 02:55:41,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:41,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:41,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:41,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:41,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:41,638 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,639 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:41,639 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:41,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146cae2f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:41,640 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:41,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,643 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,643 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:41,643 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:41,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:41,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:41,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:41,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:41,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:41,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:41,648 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:41,648 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:41,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,658 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:41,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:41,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,659 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:41,659 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:41,759 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:41,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:41,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:41,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:41,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:41,768 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:41,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:41,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:41,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,771 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,771 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:41,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:41,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:41,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:41,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:41,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:41,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:41,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:41,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:41,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:41,813 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:41,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,815 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,815 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5bd72b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:41,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:41,817 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:41,818 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:41,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:41,820 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:41,823 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:41,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:41,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:41,823 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:41,823 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:41,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,826 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,827 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:41,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,827 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,828 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,829 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:41,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:41,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:41,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:41,829 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:41,829 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,829 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681354541814, max timestamp=1681354541833Num uniques 859; 2023-04-13 02:55:41,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 02:55:41,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681354541814, max timestamp=1681354541833Num uniques 859; 2023-04-13 02:55:41,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 02:55:41,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681354541814, max timestamp=1681354541834Num uniques 859; 2023-04-13 02:55:41,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 02:55:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354601861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354601861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354601866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41016 deadline: 1681354601868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354601871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,929 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:41,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:41,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:41,945 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:41,945 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:41,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:41,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:41,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:41,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:41,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:41,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:41,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:41,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:41,978 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,978 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:41,979 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:41,979 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:41,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:41,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:41,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:41,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:41,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,981 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,982 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354601972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354601972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:41,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,984 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:41,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3acd21[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:41,984 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354601978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,985 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:41,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:41,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:41,988 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:41,988 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:41,988 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:41,988 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:41,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:41,990 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:41,990 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:41,990 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:41,990 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:41,990 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:41,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:41,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:41,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:41,995 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:41,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:41,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354601982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:41,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354601981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,002 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,002 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,002 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,003 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,003 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,003 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,003 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:42,103 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,107 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,107 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,108 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,108 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,122 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:42,125 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:42,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:42,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:42,125 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:42,126 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,126 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,127 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,128 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:42,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eabd712[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:42,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:42,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,131 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:42,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:42,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,137 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:42,138 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:42,140 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:42,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:42,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:42,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:42,140 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:42,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,143 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,146 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,151 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,152 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,152 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,152 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,152 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,152 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354602186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354602186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354602186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354602199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354602200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/bfb0e62cfe854a8e9fa3d57bb5c52204 2023-04-13 02:55:42,253 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,276 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,276 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,276 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,276 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,277 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,297 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3ec576cc36844a7499870ce93e0b0baf 2023-04-13 02:55:42,298 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,298 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,298 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:42,300 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:42,303 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,303 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,304 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:42,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f47053e[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:42,306 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:42,306 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,307 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:42,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:42,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:42,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,309 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:42,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:42,312 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:42,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:42,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:42,312 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:42,312 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:42,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,324 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,327 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,335 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,344 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,352 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,352 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,452 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,457 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:42,463 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:42,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:42,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,464 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,465 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd66f72[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:42,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:42,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,467 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:42,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:42,468 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:42,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:42,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,470 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:42,470 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:42,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:42,471 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:42,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:42,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:42,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:42,471 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:42,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,472 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,473 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,474 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,479 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,479 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,480 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,480 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,480 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,481 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354602500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354602500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41004 deadline: 1681354602500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354602510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:42,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354602513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,581 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,592 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,592 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,593 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,593 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,594 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,598 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,598 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,598 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:42,598 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:42,601 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,601 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,602 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:42,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559cf00c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:42,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:42,603 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:42,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:42,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:42,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,607 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:42,607 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:42,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:42,610 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:42,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:42,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:42,610 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:42,610 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:42,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,621 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,630 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,631 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,631 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,631 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,731 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,735 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,735 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,736 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,736 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,738 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,738 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,738 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,739 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:42,739 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:42,744 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:42,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451faeea[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:42,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:42,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:42,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:42,755 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,755 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,756 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:42,758 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:42,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:42,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:42,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:42,759 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:42,760 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,760 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,760 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:42,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:42,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,767 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:42,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:42,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:42,768 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,768 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:42,768 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:42,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:42,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/71da31c17de24de9a97418f7397b0486 2023-04-13 02:55:42,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/bfb0e62cfe854a8e9fa3d57bb5c52204 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204 2023-04-13 02:55:42,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204, entries=250, sequenceid=248, filesize=16.4 K 2023-04-13 02:55:42,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3ec576cc36844a7499870ce93e0b0baf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf 2023-04-13 02:55:42,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf, entries=150, sequenceid=248, filesize=11.8 K 2023-04-13 02:55:42,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/71da31c17de24de9a97418f7397b0486 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486 2023-04-13 02:55:42,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486, entries=150, sequenceid=248, filesize=11.8 K 2023-04-13 02:55:42,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 9a7666d97e523c69aa839344392ace09 in 1008ms, sequenceid=248, compaction requested=true 2023-04-13 02:55:42,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:42,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:42,799 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:42,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:42,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:42,804 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:42,804 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53198 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:42,804 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:42,804 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,804 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/becaaa52da194ac19be960f532653edc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=52.0 K 2023-04-13 02:55:42,805 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:42,805 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:42,806 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,806 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5150cee686f4b20ab63ad06f4d1b352, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=47.3 K 2023-04-13 02:55:42,806 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a5150cee686f4b20ab63ad06f4d1b352, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:42,806 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0aeffdc7ca8043898e7de9e17bb6881f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681354539471 2023-04-13 02:55:42,807 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9f2e639da5c74faa8835eb7fa829b47e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681354540633 2023-04-13 02:55:42,807 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting becaaa52da194ac19be960f532653edc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:42,807 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3ec576cc36844a7499870ce93e0b0baf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:42,807 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ac4da4288fbc4b71b9ddc9a0fbae3655, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681354539471 2023-04-13 02:55:42,808 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting df44d2bea8484f8f928cfd7c3fc14981, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681354540633 2023-04-13 02:55:42,808 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bfb0e62cfe854a8e9fa3d57bb5c52204, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:42,829 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:42,837 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#412 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:42,868 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/3b709bee30854310a60bcd8af6090512 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/3b709bee30854310a60bcd8af6090512 2023-04-13 02:55:42,868 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:42,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:42,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:42,872 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into 3b709bee30854310a60bcd8af6090512(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:42,872 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:42,872 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=12, startTime=1681354542799; duration=0sec 2023-04-13 02:55:42,872 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:42,873 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:42,874 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:42,874 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:42,874 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,874 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4caf43e521ee4c41a29bb83d83ef5f93, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=47.3 K 2023-04-13 02:55:42,874 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4caf43e521ee4c41a29bb83d83ef5f93, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681354539121 2023-04-13 02:55:42,875 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f419362c156241e495ed009e573cd9e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681354539471 2023-04-13 02:55:42,875 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 19da798adfa54613b2ab894421e618b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681354540633 2023-04-13 02:55:42,876 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 71da31c17de24de9a97418f7397b0486, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:42,878 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/62867cdb2b06452ab89da509b31178de as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/62867cdb2b06452ab89da509b31178de 2023-04-13 02:55:42,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:42,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:42,884 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:42,884 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,884 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:42,884 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:42,885 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 62867cdb2b06452ab89da509b31178de(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:42,885 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:42,885 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=12, startTime=1681354542799; duration=0sec 2023-04-13 02:55:42,885 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:42,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,887 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#413 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:42,887 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:42,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:42,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:42,888 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:42,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,888 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:42,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:42,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:42,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:42,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:42,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:42,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:42,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:42,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:42,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:42,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:42,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:42,908 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-13 02:55:42,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:42,909 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:42,909 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:42,909 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:42,909 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:42,909 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:42,937 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0ef6fb5db00b431199108ca01e8b82ee as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ef6fb5db00b431199108ca01e8b82ee 2023-04-13 02:55:42,943 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into 0ef6fb5db00b431199108ca01e8b82ee(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:42,943 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:42,944 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=12, startTime=1681354542800; duration=0sec 2023-04-13 02:55:42,944 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:42,948 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/dbfada21032b44e192709885ead364bc 2023-04-13 02:55:42,993 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1613d0c187e8438ba6b6dffe4556fcdb 2023-04-13 02:55:43,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:43,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,040 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/93f3af59b19a43ea8e38f0c55e392e4f 2023-04-13 02:55:43,043 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/dbfada21032b44e192709885ead364bc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc 2023-04-13 02:55:43,046 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc, entries=150, sequenceid=285, filesize=11.9 K 2023-04-13 02:55:43,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1613d0c187e8438ba6b6dffe4556fcdb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb 2023-04-13 02:55:43,060 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb, entries=150, sequenceid=285, filesize=11.9 K 2023-04-13 02:55:43,061 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/93f3af59b19a43ea8e38f0c55e392e4f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f 2023-04-13 02:55:43,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354603038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,071 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f, entries=150, sequenceid=285, filesize=11.9 K 2023-04-13 02:55:43,072 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=60.38 KB/61830 for 9a7666d97e523c69aa839344392ace09 in 164ms, sequenceid=285, compaction requested=false 2023-04-13 02:55:43,072 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,073 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:43,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 02:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:43,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:43,073 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,075 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,076 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:43,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42762d82[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:43,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:43,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,079 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:43,079 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:43,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:43,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:43,085 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:43,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:43,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:43,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:43,085 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:43,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,091 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,093 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,095 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,096 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:43,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:43,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:43,096 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:43,096 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:43,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f8d567cd45164e338b10d1d71025100f 2023-04-13 02:55:43,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3c5a29b21ddb4bdaa93fba68d187249b 2023-04-13 02:55:43,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/8c2ac58d551d4251916082c116ffdc09 2023-04-13 02:55:43,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f8d567cd45164e338b10d1d71025100f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f 2023-04-13 02:55:43,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f, entries=150, sequenceid=299, filesize=11.9 K 2023-04-13 02:55:43,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/3c5a29b21ddb4bdaa93fba68d187249b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b 2023-04-13 02:55:43,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b, entries=100, sequenceid=299, filesize=9.5 K 2023-04-13 02:55:43,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/8c2ac58d551d4251916082c116ffdc09 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09 2023-04-13 02:55:43,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09, entries=100, sequenceid=299, filesize=9.5 K 2023-04-13 02:55:43,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 88ms, sequenceid=299, compaction requested=true 2023-04-13 02:55:43,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:43,161 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:43,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:43,161 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:43,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:43,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:43,162 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:43,162 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,164 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36814 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:43,164 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:43,164 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,164 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/3b709bee30854310a60bcd8af6090512, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=36.0 K 2023-04-13 02:55:43,165 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34374 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:43,165 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3b709bee30854310a60bcd8af6090512, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:43,165 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:43,165 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,165 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/62867cdb2b06452ab89da509b31178de, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=33.6 K 2023-04-13 02:55:43,165 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 62867cdb2b06452ab89da509b31178de, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:43,165 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dbfada21032b44e192709885ead364bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681354541859 2023-04-13 02:55:43,166 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1613d0c187e8438ba6b6dffe4556fcdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681354541859 2023-04-13 02:55:43,166 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f8d567cd45164e338b10d1d71025100f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354543040 2023-04-13 02:55:43,166 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3c5a29b21ddb4bdaa93fba68d187249b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354543040 2023-04-13 02:55:43,196 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:43,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:43,202 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:43,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:43,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:43,202 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,204 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:43,205 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,205 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,205 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,205 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:43,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:43,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:43,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,208 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:43,210 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:43,211 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:43,211 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:43,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:43,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,213 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:43,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:43,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354603202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354603203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354603203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354603204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354603204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:43,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:43,222 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,222 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,223 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,223 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:43,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d246741[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:43,233 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:43,233 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:43,236 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:43,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:43,241 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:43,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:43,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:43,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:43,242 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:43,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:43,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:43,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,254 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:43,254 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:43,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,274 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/a0ad913376b64fe9b28d6c26d9cfa3d7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a0ad913376b64fe9b28d6c26d9cfa3d7 2023-04-13 02:55:43,285 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into a0ad913376b64fe9b28d6c26d9cfa3d7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:43,285 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,285 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=13, startTime=1681354543161; duration=0sec 2023-04-13 02:55:43,285 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:43,285 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:43,286 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34374 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:43,286 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:43,286 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,286 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ef6fb5db00b431199108ca01e8b82ee, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=33.6 K 2023-04-13 02:55:43,287 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 0ef6fb5db00b431199108ca01e8b82ee, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1681354541781 2023-04-13 02:55:43,287 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 93f3af59b19a43ea8e38f0c55e392e4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681354541859 2023-04-13 02:55:43,287 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8c2ac58d551d4251916082c116ffdc09, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354543040 2023-04-13 02:55:43,297 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:43,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354603316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354603317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354603317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354603317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354603317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,332 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/ac0b66e80b7b490a97cdddefb81b2ba0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ac0b66e80b7b490a97cdddefb81b2ba0 2023-04-13 02:55:43,337 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into ac0b66e80b7b490a97cdddefb81b2ba0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:43,338 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,338 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=13, startTime=1681354543161; duration=0sec 2023-04-13 02:55:43,338 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:43,354 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:43,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:43,358 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:43,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:43,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:43,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,360 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:43,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:43,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:43,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:43,361 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:43,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:43,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:43,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:43,364 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8aec79[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:43,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:43,366 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,367 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:43,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:43,370 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:43,373 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:43,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:43,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:43,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:43,373 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:43,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,375 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,375 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:43,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:43,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:43,381 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:43,381 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,481 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:43,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:43,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:43,485 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:43,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:43,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:43,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,486 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,487 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,487 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:43,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:43,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:43,490 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,490 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:43,490 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,490 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:43,490 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,493 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37da6b9c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:43,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:43,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,497 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:43,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:43,501 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:43,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:43,503 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:43,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:43,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:43,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:43,504 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:43,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,513 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:43,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:43,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,513 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:43,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,513 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,513 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:43,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354603521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354603523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354603523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354603523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354603524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/8d9f1846788e4d6887aab1f56670531b 2023-04-13 02:55:43,614 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:43,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:43,627 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:43,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:43,627 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:43,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,630 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,630 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,630 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:43,634 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:43,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:43,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,646 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:43,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:43,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,652 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:43,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/5d9f8101a69a45b2ad57a07abbcd8fa6 2023-04-13 02:55:43,653 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,653 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:43,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,655 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:43,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b35ad8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:43,658 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:43,658 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,660 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,660 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,660 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:43,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:43,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:43,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:43,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:43,670 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:43,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:43,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:43,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:43,671 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:43,671 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/30bfa92b6f9c4920aa12a4944c49c8cf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/30bfa92b6f9c4920aa12a4944c49c8cf 2023-04-13 02:55:43,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,676 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:43,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:43,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,681 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 30bfa92b6f9c4920aa12a4944c49c8cf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:43,681 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,681 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354543161; duration=0sec 2023-04-13 02:55:43,681 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:43,682 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:43,686 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,687 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:43,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:43,687 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:43,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:43,688 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:43,688 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:43,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:43,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/cfb8002c5cd5453994cbaa6080fba3b8 2023-04-13 02:55:43,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/8d9f1846788e4d6887aab1f56670531b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b 2023-04-13 02:55:43,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 02:55:43,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/5d9f8101a69a45b2ad57a07abbcd8fa6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6 2023-04-13 02:55:43,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 02:55:43,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/cfb8002c5cd5453994cbaa6080fba3b8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8 2023-04-13 02:55:43,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8, entries=150, sequenceid=316, filesize=11.9 K 2023-04-13 02:55:43,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 566ms, sequenceid=316, compaction requested=false 2023-04-13 02:55:43,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:43,788 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:43,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:43,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:43,791 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:43,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:43,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:43,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:43,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:43,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:43,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:43,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:43,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:43,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:43,800 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:43,800 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:55:43,800 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:43,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:43,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:43,801 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:43,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:43,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:43,840 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/882dd932c2864e4d9043f9a78646f770 2023-04-13 02:55:43,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354603910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354603910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354603911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354603912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:43,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354603919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354604021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354604021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354604022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354604022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354604029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354604224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354604225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354604225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354604225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354604235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,256 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/e4884ff94d714019b30fabb64e8997a4 2023-04-13 02:55:44,282 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/17ea5121a0814432be232d27bd70a1af 2023-04-13 02:55:44,289 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/882dd932c2864e4d9043f9a78646f770 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770 2023-04-13 02:55:44,292 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770, entries=150, sequenceid=335, filesize=11.9 K 2023-04-13 02:55:44,293 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/e4884ff94d714019b30fabb64e8997a4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4 2023-04-13 02:55:44,299 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4, entries=150, sequenceid=335, filesize=11.9 K 2023-04-13 02:55:44,306 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/17ea5121a0814432be232d27bd70a1af as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af 2023-04-13 02:55:44,319 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af, entries=150, sequenceid=335, filesize=11.9 K 2023-04-13 02:55:44,331 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 531ms, sequenceid=335, compaction requested=true 2023-04-13 02:55:44,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:44,331 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:44,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,333 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:44,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab33360[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:44,335 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:44,335 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:44,336 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,336 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:44,338 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:44,340 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:44,340 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:44,340 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:44,341 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:44,341 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:44,343 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,343 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,343 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,344 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,345 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:44,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,348 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:44,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:44,348 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:44,348 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:44,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:44,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:44,448 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:44,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:44,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:44,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:44,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:44,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:44,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,455 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:44,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:44,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:44,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:44,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:44,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:44,460 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:44,460 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:44,497 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f111dd44968740939cfb755149e07945 2023-04-13 02:55:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:44,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:44,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354604549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354604552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354604553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354604554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354604555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354604656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354604665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354604665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354604665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354604665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354604867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354604867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354604868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354604869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354604869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,914 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/d935c1a385b94f5eb2823c1247b44a0f 2023-04-13 02:55:44,925 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0aa01024e4994421a409ab479017754a 2023-04-13 02:55:44,929 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/f111dd44968740939cfb755149e07945 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945 2023-04-13 02:55:44,933 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945, entries=150, sequenceid=351, filesize=11.9 K 2023-04-13 02:55:44,934 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/d935c1a385b94f5eb2823c1247b44a0f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f 2023-04-13 02:55:44,937 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f, entries=150, sequenceid=351, filesize=11.9 K 2023-04-13 02:55:44,937 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0aa01024e4994421a409ab479017754a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a 2023-04-13 02:55:44,944 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a, entries=150, sequenceid=351, filesize=11.9 K 2023-04-13 02:55:44,945 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 485ms, sequenceid=351, compaction requested=true 2023-04-13 02:55:44,945 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:44,945 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:44,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:44,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:44,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:44,945 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:44,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,947 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:44,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282e56b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:44,948 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:44,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:44,951 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:44,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,951 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:44,952 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:44,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:44,953 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:44,954 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:44,954 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:44,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:44,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:44,955 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:44,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:44,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:44,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:44,957 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:44,958 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:44,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:44,960 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,961 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:44,961 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:44,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:44,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:44,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:44,960 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:44,962 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:44,962 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:45,062 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:45,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:45,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:45,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:45,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:45,066 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:45,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:45,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:45,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:45,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:45,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:45,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:45,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:45,070 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:45,070 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:45,071 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:45,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,086 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/bebb4d3916244faebf79cba3aa0db656 2023-04-13 02:55:45,102 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/2d6840c1ca5c41fc9d4aa777b42b3f1b 2023-04-13 02:55:45,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:45,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354605196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354605198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354605199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354605200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354605202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354605304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354605310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354605311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354605311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354605311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354605509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,515 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/32cae96135ad4cf6afd61e0e080e41c7 2023-04-13 02:55:45,519 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/bebb4d3916244faebf79cba3aa0db656 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656 2023-04-13 02:55:45,522 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656, entries=150, sequenceid=367, filesize=11.9 K 2023-04-13 02:55:45,523 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/2d6840c1ca5c41fc9d4aa777b42b3f1b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b 2023-04-13 02:55:45,526 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b, entries=150, sequenceid=367, filesize=11.9 K 2023-04-13 02:55:45,526 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/32cae96135ad4cf6afd61e0e080e41c7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7 2023-04-13 02:55:45,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354605516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354605516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354605516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354605517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,530 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7, entries=150, sequenceid=367, filesize=11.9 K 2023-04-13 02:55:45,530 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a7666d97e523c69aa839344392ace09 in 459ms, sequenceid=367, compaction requested=true 2023-04-13 02:55:45,530 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:45,530 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:45,532 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:45,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6660e82c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:45,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:45,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:45,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:45,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:45,541 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:45,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:45,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:45,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:45,541 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:45,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:45,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:45,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:45,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:45,548 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,548 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:45,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:45,548 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:45,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:45,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,648 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:45,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:45,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:45,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:45,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:45,652 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:45,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:45,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:45,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:45,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,654 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:45,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:45,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:45,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:45,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:45,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:45,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:45,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:45,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:45,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:45,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:45,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:45,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:45,657 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:55:45,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:45,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:45,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:45,658 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:45,668 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/341d585cf7834720ac8ef09a7faa118a 2023-04-13 02:55:45,678 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/69f386c611ec4ab6889aaf322bfb72df 2023-04-13 02:55:45,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:45,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:45,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354605851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354605854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354605854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354605855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354605855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354605961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354605963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354605963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354605969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:45,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:45,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354605969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,087 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/28fedda3ec7144f88178be81249dd085 2023-04-13 02:55:46,092 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/341d585cf7834720ac8ef09a7faa118a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a 2023-04-13 02:55:46,095 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a, entries=150, sequenceid=383, filesize=11.9 K 2023-04-13 02:55:46,096 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/69f386c611ec4ab6889aaf322bfb72df as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df 2023-04-13 02:55:46,110 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df, entries=150, sequenceid=383, filesize=11.9 K 2023-04-13 02:55:46,111 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/28fedda3ec7144f88178be81249dd085 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085 2023-04-13 02:55:46,117 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085, entries=150, sequenceid=383, filesize=11.9 K 2023-04-13 02:55:46,117 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 460ms, sequenceid=383, compaction requested=true 2023-04-13 02:55:46,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:46,117 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,117 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,124 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385b1ba2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,130 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,130 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,131 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,131 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,132 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,134 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,134 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,134 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,135 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,136 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,136 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,138 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,140 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,140 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:46,140 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,140 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,141 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,141 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:46,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:55:46,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:46,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:46,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:46,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/304374ba7eb74b6daa7cb1b1e0bf723b 2023-04-13 02:55:46,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354606190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354606194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354606199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354606200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354606200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,241 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:46,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:46,244 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:46,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:46,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:46,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,247 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,247 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:46,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:46,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,248 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:46,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,250 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b3007fa[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,252 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,252 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,253 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,255 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,257 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,257 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,260 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,260 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,260 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,262 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,264 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,265 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,265 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354606301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354606308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354606310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354606310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354606310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,365 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:46,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:46,368 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:46,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:46,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:46,369 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,369 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,370 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:46,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:46,372 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8c1c4a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,382 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,382 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,383 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,386 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,387 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,390 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,391 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,393 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:46,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,393 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,393 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,396 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,396 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,493 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:46,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:46,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:46,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:46,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:46,497 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:46,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:46,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:46,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,504 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,504 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,505 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:46,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:46,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,508 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fccb83f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,512 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,512 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354606509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,514 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,517 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,518 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,518 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,518 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,518 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,520 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,520 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,521 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354606517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354606517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354606518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354606518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,524 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,524 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,524 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,525 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,525 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,525 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/b2af266c9a6e4ec5a44d3f89eee37cd6 2023-04-13 02:55:46,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0ab8e2d1e5644cbc84406a218ae82d6f 2023-04-13 02:55:46,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/304374ba7eb74b6daa7cb1b1e0bf723b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b 2023-04-13 02:55:46,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b, entries=200, sequenceid=401, filesize=14.3 K 2023-04-13 02:55:46,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/b2af266c9a6e4ec5a44d3f89eee37cd6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6 2023-04-13 02:55:46,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6, entries=150, sequenceid=401, filesize=11.9 K 2023-04-13 02:55:46,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0ab8e2d1e5644cbc84406a218ae82d6f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f 2023-04-13 02:55:46,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f, entries=150, sequenceid=401, filesize=11.9 K 2023-04-13 02:55:46,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 9a7666d97e523c69aa839344392ace09 in 446ms, sequenceid=401, compaction requested=true 2023-04-13 02:55:46,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:46,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:46,621 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:55:46,621 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 02:55:46,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:46,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:46,625 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:46,626 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88254 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:55:46,626 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:46,626 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,626 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a0ad913376b64fe9b28d6c26d9cfa3d7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=86.2 K 2023-04-13 02:55:46,627 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85814 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 02:55:46,627 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:46,627 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,627 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/30bfa92b6f9c4920aa12a4944c49c8cf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=83.8 K 2023-04-13 02:55:46,627 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 30bfa92b6f9c4920aa12a4944c49c8cf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354541865 2023-04-13 02:55:46,627 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a0ad913376b64fe9b28d6c26d9cfa3d7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354541865 2023-04-13 02:55:46,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5d9f8101a69a45b2ad57a07abbcd8fa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681354543137 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8d9f1846788e4d6887aab1f56670531b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681354543137 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e4884ff94d714019b30fabb64e8997a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681354543189 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 882dd932c2864e4d9043f9a78646f770, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681354543189 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d935c1a385b94f5eb2823c1247b44a0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681354543909 2023-04-13 02:55:46,628 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f111dd44968740939cfb755149e07945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681354543909 2023-04-13 02:55:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:46,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:46,629 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:46,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,629 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2d6840c1ca5c41fc9d4aa777b42b3f1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681354544549 2023-04-13 02:55:46,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:46,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:46,629 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting bebb4d3916244faebf79cba3aa0db656, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681354544549 2023-04-13 02:55:46,629 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 69f386c611ec4ab6889aaf322bfb72df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1681354545198 2023-04-13 02:55:46,629 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 341d585cf7834720ac8ef09a7faa118a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1681354545198 2023-04-13 02:55:46,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,629 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,629 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b2af266c9a6e4ec5a44d3f89eee37cd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681354545838 2023-04-13 02:55:46,630 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 304374ba7eb74b6daa7cb1b1e0bf723b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681354545838 2023-04-13 02:55:46,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,631 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,632 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,632 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:46,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:46,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:46,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:46,634 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,634 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:46,634 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 02:55:46,635 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:46,635 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,635 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:46,636 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,645 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:46,650 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#443 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:46,661 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/93c040921cb34e31b3b3986c9b0b627c 2023-04-13 02:55:46,706 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/dd35d95045a641bd83fde81f233f5a8b 2023-04-13 02:55:46,726 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/398e9e9975ee41cb998a728021a3cfd7 2023-04-13 02:55:46,729 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/93c040921cb34e31b3b3986c9b0b627c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c 2023-04-13 02:55:46,732 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 02:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/dd35d95045a641bd83fde81f233f5a8b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b 2023-04-13 02:55:46,736 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 02:55:46,737 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/398e9e9975ee41cb998a728021a3cfd7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7 2023-04-13 02:55:46,740 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 02:55:46,741 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 9a7666d97e523c69aa839344392ace09 in 107ms, sequenceid=415, compaction requested=true 2023-04-13 02:55:46,741 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:46,741 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,744 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,747 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,747 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6faa346f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,750 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,750 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,752 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,753 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,759 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,759 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,759 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,762 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,766 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,771 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,771 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,771 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:46,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:46,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:46,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:46,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:46,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:46,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/2ff0013674224c31bce67fc8f1f0f5a8 2023-04-13 02:55:46,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/8ae60bbf3da847a9a795f284019d6bbc 2023-04-13 02:55:46,871 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:46,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:46,877 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:46,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:46,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:46,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,877 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,879 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,879 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,879 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,879 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:46,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:46,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:46,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:46,884 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:46,885 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:46,889 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145ef1b7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:46,891 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:46,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,893 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:46,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,894 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:46,896 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:46,899 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:46,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:46,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:46,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:46,899 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:46,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354546858, max timestamp=1681354546898Num uniques 859; 2023-04-13 02:55:46,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354546858, max timestamp=1681354546897Num uniques 859; 2023-04-13 02:55:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354546858, max timestamp=1681354546898Num uniques 859; 2023-04-13 02:55:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:46,910 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,910 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:46,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,910 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:46,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:46,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:46,914 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:46,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:46,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:46,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:46,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:46,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:46,915 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:46,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:46,915 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:46,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:46,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354606943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354606943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354606944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354606946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:46,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354606949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,015 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:47,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:47,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:47,019 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:47,019 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,019 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:47,019 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:47,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,021 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:47,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:47,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:47,023 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,023 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:47,024 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:47,024 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:47,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:47,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3ed13a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:47,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:47,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:47,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:47,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:47,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:47,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:47,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:47,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:47,037 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:47,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,044 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:47,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:47,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,049 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:47,049 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:47,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:47,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354607056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354607056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354607059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354607059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354607060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,071 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/6d7977006cd0483ab34c24f0d1fe5189 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/6d7977006cd0483ab34c24f0d1fe5189 2023-04-13 02:55:47,076 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 6d7977006cd0483ab34c24f0d1fe5189(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,076 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,076 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=9, startTime=1681354546621; duration=0sec 2023-04-13 02:55:47,076 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,076 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-13 02:55:47,078 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98000 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-13 02:55:47,078 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:47,078 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,078 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ac0b66e80b7b490a97cdddefb81b2ba0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=95.7 K 2023-04-13 02:55:47,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ac0b66e80b7b490a97cdddefb81b2ba0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681354541865 2023-04-13 02:55:47,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cfb8002c5cd5453994cbaa6080fba3b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681354543137 2023-04-13 02:55:47,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 17ea5121a0814432be232d27bd70a1af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681354543189 2023-04-13 02:55:47,079 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0aa01024e4994421a409ab479017754a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681354543909 2023-04-13 02:55:47,080 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 32cae96135ad4cf6afd61e0e080e41c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681354544549 2023-04-13 02:55:47,080 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 28fedda3ec7144f88178be81249dd085, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1681354545198 2023-04-13 02:55:47,081 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0ab8e2d1e5644cbc84406a218ae82d6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681354545838 2023-04-13 02:55:47,081 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 398e9e9975ee41cb998a728021a3cfd7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681354546198 2023-04-13 02:55:47,084 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/836891b045484748bcda56dfdd804cf6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/836891b045484748bcda56dfdd804cf6 2023-04-13 02:55:47,088 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into 836891b045484748bcda56dfdd804cf6(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,088 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,088 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=9, startTime=1681354546621; duration=0sec 2023-04-13 02:55:47,088 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:47,094 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:47,101 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/d51f078b28084b06a0621263ab8c4abc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/d51f078b28084b06a0621263ab8c4abc 2023-04-13 02:55:47,105 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into d51f078b28084b06a0621263ab8c4abc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,105 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,105 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=8, startTime=1681354546621; duration=0sec 2023-04-13 02:55:47,105 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:47,150 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:47,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:47,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:47,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:47,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:47,153 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:47,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:47,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:47,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,155 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:47,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:47,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:47,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:47,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:47,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:47,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,158 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:47,158 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:47,158 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:47,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:47,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:47,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70599f02[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:47,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:47,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:47,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,166 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:47,169 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:47,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:47,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:47,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:47,169 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:47,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,174 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,176 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:47,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,176 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,177 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:47,177 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:47,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354607265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354607265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354607268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354607268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354607269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,277 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:47,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:47,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:47,280 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,280 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:47,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:47,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:47,285 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:47,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:47,286 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:47,286 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:47,287 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,288 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:47,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:47,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eeed42a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:47,289 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0ed05fd8e13d4f1086e4695da70566a3 2023-04-13 02:55:47,290 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:47,292 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:47,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:47,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:47,293 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:47,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:47,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:47,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:47,293 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:47,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/2ff0013674224c31bce67fc8f1f0f5a8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8 2023-04-13 02:55:47,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8, entries=200, sequenceid=425, filesize=14.3 K 2023-04-13 02:55:47,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/8ae60bbf3da847a9a795f284019d6bbc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc 2023-04-13 02:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,313 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,316 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc, entries=100, sequenceid=425, filesize=9.5 K 2023-04-13 02:55:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:47,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:47,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:47,317 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/0ed05fd8e13d4f1086e4695da70566a3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3 2023-04-13 02:55:47,317 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:47,317 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:47,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3, entries=100, sequenceid=425, filesize=9.5 K 2023-04-13 02:55:47,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9a7666d97e523c69aa839344392ace09 in 487ms, sequenceid=425, compaction requested=true 2023-04-13 02:55:47,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:47,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,322 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:47,322 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:47,322 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39744 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:47,323 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:47,323 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/836891b045484748bcda56dfdd804cf6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=38.8 K 2023-04-13 02:55:47,323 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,323 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/6d7977006cd0483ab34c24f0d1fe5189, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=34.0 K 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 836891b045484748bcda56dfdd804cf6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681354545838 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6d7977006cd0483ab34c24f0d1fe5189, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681354545838 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 93c040921cb34e31b3b3986c9b0b627c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681354546198 2023-04-13 02:55:47,323 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting dd35d95045a641bd83fde81f233f5a8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681354546198 2023-04-13 02:55:47,324 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2ff0013674224c31bce67fc8f1f0f5a8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681354546830 2023-04-13 02:55:47,324 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8ae60bbf3da847a9a795f284019d6bbc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681354546830 2023-04-13 02:55:47,329 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#451 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:47,330 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:47,341 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/a5779eab03f04195b7e871d9d5e5a2c5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5779eab03f04195b7e871d9d5e5a2c5 2023-04-13 02:55:47,347 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into a5779eab03f04195b7e871d9d5e5a2c5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,347 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354547322; duration=0sec 2023-04-13 02:55:47,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 02:55:47,348 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:47,348 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:47,348 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:47,417 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:47,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:47,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:47,421 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:47,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:47,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:47,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,422 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,423 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:47,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:47,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,424 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:47,424 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:47,425 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:47,436 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/c3840c9b4a0e4995a12bc7843764e76d 2023-04-13 02:55:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:47,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:47,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354607578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354607578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354607581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354607582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354607583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354607688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354607688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354607690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354607690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354607691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,739 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/c696d45a83174a1fae1cc4f9d41034b0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c696d45a83174a1fae1cc4f9d41034b0 2023-04-13 02:55:47,749 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into c696d45a83174a1fae1cc4f9d41034b0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,749 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,749 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=13, startTime=1681354547321; duration=0sec 2023-04-13 02:55:47,749 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:47,847 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/cef1f22471be43779d2ddf3c918501ad 2023-04-13 02:55:47,857 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/dd5b1a8806374449bf547586b260cc28 2023-04-13 02:55:47,861 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/c3840c9b4a0e4995a12bc7843764e76d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d 2023-04-13 02:55:47,865 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d, entries=150, sequenceid=465, filesize=11.9 K 2023-04-13 02:55:47,866 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/cef1f22471be43779d2ddf3c918501ad as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad 2023-04-13 02:55:47,870 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad, entries=150, sequenceid=465, filesize=11.9 K 2023-04-13 02:55:47,871 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/dd5b1a8806374449bf547586b260cc28 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28 2023-04-13 02:55:47,875 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28, entries=150, sequenceid=465, filesize=11.9 K 2023-04-13 02:55:47,876 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 9a7666d97e523c69aa839344392ace09 in 452ms, sequenceid=465, compaction requested=true 2023-04-13 02:55:47,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:47,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:47,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:47,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:47,879 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,879 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,879 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,879 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dde5709[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:47,880 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:47,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,883 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:47,884 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,884 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:47,885 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:47,890 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:47,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:47,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:47,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:47,891 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:47,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:47,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:47,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:55:47,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:47,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:47,901 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:47,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:47,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:47,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:47,901 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:47,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,902 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,902 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:47,902 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:47,901 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:47,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:47,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:47,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/db6ca83148fb4d7d89e0271c264fbd6e 2023-04-13 02:55:47,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4b331629a5c04f1a9290a70d7e43575f 2023-04-13 02:55:47,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/7993535bffa24f878df8375defc40e3b 2023-04-13 02:55:47,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354607922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354607923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354607923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354607924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:47,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354607925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:47,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/db6ca83148fb4d7d89e0271c264fbd6e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e 2023-04-13 02:55:47,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e, entries=200, sequenceid=478, filesize=14.3 K 2023-04-13 02:55:47,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4b331629a5c04f1a9290a70d7e43575f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f 2023-04-13 02:55:47,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f, entries=150, sequenceid=478, filesize=11.9 K 2023-04-13 02:55:47,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/7993535bffa24f878df8375defc40e3b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b 2023-04-13 02:55:47,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b, entries=150, sequenceid=478, filesize=11.9 K 2023-04-13 02:55:47,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9a7666d97e523c69aa839344392ace09 in 51ms, sequenceid=478, compaction requested=true 2023-04-13 02:55:47,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:47,949 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:47,949 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:47,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,950 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:47,950 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:47,950 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:47,950 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,950 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:47,950 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c696d45a83174a1fae1cc4f9d41034b0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=38.9 K 2023-04-13 02:55:47,950 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,951 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5779eab03f04195b7e871d9d5e5a2c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=36.5 K 2023-04-13 02:55:47,951 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c696d45a83174a1fae1cc4f9d41034b0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681354546199 2023-04-13 02:55:47,951 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a5779eab03f04195b7e871d9d5e5a2c5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681354546199 2023-04-13 02:55:47,951 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cef1f22471be43779d2ddf3c918501ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681354546942 2023-04-13 02:55:47,951 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c3840c9b4a0e4995a12bc7843764e76d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681354546942 2023-04-13 02:55:47,951 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4b331629a5c04f1a9290a70d7e43575f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:47,952 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting db6ca83148fb4d7d89e0271c264fbd6e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:47,957 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:47,958 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:47,974 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/c7b444a2efd44112944489926ca06a64 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/c7b444a2efd44112944489926ca06a64 2023-04-13 02:55:47,978 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into c7b444a2efd44112944489926ca06a64(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:47,978 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:47,978 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354547949; duration=0sec 2023-04-13 02:55:47,979 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:47,979 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:47,980 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:47,980 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:47,980 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:47,980 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/d51f078b28084b06a0621263ab8c4abc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=46.0 K 2023-04-13 02:55:47,980 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d51f078b28084b06a0621263ab8c4abc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681354546198 2023-04-13 02:55:47,980 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0ed05fd8e13d4f1086e4695da70566a3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681354546830 2023-04-13 02:55:47,981 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting dd5b1a8806374449bf547586b260cc28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681354546942 2023-04-13 02:55:47,981 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7993535bffa24f878df8375defc40e3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:47,987 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#460 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:48,002 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:48,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:48,009 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:48,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:48,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:48,009 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:48,010 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,010 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,012 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:48,012 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:48,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,013 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:48,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:48,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:48,014 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:48,014 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:48,014 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:48,015 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:48,015 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:48,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:48,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:48,039 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/27e961eb75b34dbca7d37bcb36cb29fc 2023-04-13 02:55:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354608042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354608043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354608046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354608047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354608051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354608152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354608152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354608154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354608155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354608155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354608358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354608358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354608361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354608361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354608362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,372 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/a6ab3e6428b34e989e43ee98a931a1b9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a6ab3e6428b34e989e43ee98a931a1b9 2023-04-13 02:55:48,377 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into a6ab3e6428b34e989e43ee98a931a1b9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:48,378 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:48,378 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=13, startTime=1681354547949; duration=0sec 2023-04-13 02:55:48,378 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:48,401 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e6c33fd81604429782badc6e13ccf34b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e6c33fd81604429782badc6e13ccf34b 2023-04-13 02:55:48,404 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into e6c33fd81604429782badc6e13ccf34b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:48,405 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:48,405 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=12, startTime=1681354547949; duration=0sec 2023-04-13 02:55:48,405 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:48,449 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4eede0ec6e884154a674139978b30610 2023-04-13 02:55:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354608664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354608664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354608665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354608665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354608665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,860 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/6d760376451b4e1aba95da462d5ec21d 2023-04-13 02:55:48,864 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/27e961eb75b34dbca7d37bcb36cb29fc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc 2023-04-13 02:55:48,867 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc, entries=150, sequenceid=499, filesize=11.9 K 2023-04-13 02:55:48,868 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/4eede0ec6e884154a674139978b30610 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610 2023-04-13 02:55:48,871 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610, entries=150, sequenceid=499, filesize=11.9 K 2023-04-13 02:55:48,872 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/6d760376451b4e1aba95da462d5ec21d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d 2023-04-13 02:55:48,875 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d, entries=150, sequenceid=499, filesize=11.9 K 2023-04-13 02:55:48,875 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9a7666d97e523c69aa839344392ace09 in 860ms, sequenceid=499, compaction requested=false 2023-04-13 02:55:48,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:48,876 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:48,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:48,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:48,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:48,876 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:48,877 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,877 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4185e489[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:48,879 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:48,879 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,880 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:48,881 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:48,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,881 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:48,883 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:48,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:48,886 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:48,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:48,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:48,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:48,886 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:48,887 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,888 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,888 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:48,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:48,888 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,889 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:48,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:48,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:48,891 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:48,892 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:48,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:48,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:48,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:48,892 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,892 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:48,993 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:48,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:48,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:48,996 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:48,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:48,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:48,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:48,996 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,997 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,998 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,998 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:48,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:48,998 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:48,998 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:48,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,000 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,000 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,000 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,001 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:49,001 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,011 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b 2023-04-13 02:55:49,023 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/623f8b2148524ecbb3d0042ae54c3562 2023-04-13 02:55:49,036 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/af1c0392f4154f51a9a7f5aac50247bc 2023-04-13 02:55:49,042 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b 2023-04-13 02:55:49,046 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b, entries=150, sequenceid=513, filesize=11.9 K 2023-04-13 02:55:49,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/623f8b2148524ecbb3d0042ae54c3562 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562 2023-04-13 02:55:49,051 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562, entries=150, sequenceid=513, filesize=11.9 K 2023-04-13 02:55:49,052 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/af1c0392f4154f51a9a7f5aac50247bc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc 2023-04-13 02:55:49,057 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc, entries=150, sequenceid=513, filesize=11.9 K 2023-04-13 02:55:49,058 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 9a7666d97e523c69aa839344392ace09 in 57ms, sequenceid=513, compaction requested=true 2023-04-13 02:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,060 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d98d234[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,066 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,071 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,071 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,073 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,080 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,080 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,080 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:49,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:49,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,180 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,186 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/1e2721d1a41947ec9f4523660ec8f3b3 2023-04-13 02:55:49,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:49,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456f1629[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,196 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,196 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,196 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,198 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,201 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,201 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/ef499e4d8cf24bfe8dcef735cda0e595 2023-04-13 02:55:49,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,202 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,205 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,211 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,211 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,211 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,211 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/cd88679726f741348001232f465eed3e 2023-04-13 02:55:49,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/1e2721d1a41947ec9f4523660ec8f3b3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3 2023-04-13 02:55:49,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3, entries=350, sequenceid=523, filesize=21.4 K 2023-04-13 02:55:49,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/ef499e4d8cf24bfe8dcef735cda0e595 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354549203, max timestamp=1681354549220Num uniques 859; 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354549203, max timestamp=1681354549220Num uniques 859; 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681354549203, max timestamp=1681354549220Num uniques 859; 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:55:49,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:55:49,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595, entries=150, sequenceid=523, filesize=11.9 K 2023-04-13 02:55:49,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/cd88679726f741348001232f465eed3e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e 2023-04-13 02:55:49,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e, entries=150, sequenceid=523, filesize=11.9 K 2023-04-13 02:55:49,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 9a7666d97e523c69aa839344392ace09 in 61ms, sequenceid=523, compaction requested=true 2023-04-13 02:55:49,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:49,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,240 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:49,240 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:49,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:49,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=80.51 KB heapSize=216.23 KB 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:49,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:49,247 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:49,247 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59462 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:49,247 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:49,247 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:49,247 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,247 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,247 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/c7b444a2efd44112944489926ca06a64, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=48.5 K 2023-04-13 02:55:49,247 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a6ab3e6428b34e989e43ee98a931a1b9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=58.1 K 2023-04-13 02:55:49,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c7b444a2efd44112944489926ca06a64, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:49,248 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a6ab3e6428b34e989e43ee98a931a1b9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:49,248 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4eede0ec6e884154a674139978b30610, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681354547920 2023-04-13 02:55:49,248 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 27e961eb75b34dbca7d37bcb36cb29fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681354547920 2023-04-13 02:55:49,249 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ce4ebe0ed24c4665b6ab742d7b7d8d8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1681354548037 2023-04-13 02:55:49,249 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 623f8b2148524ecbb3d0042ae54c3562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1681354548037 2023-04-13 02:55:49,249 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 1e2721d1a41947ec9f4523660ec8f3b3, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681354549171 2023-04-13 02:55:49,249 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ef499e4d8cf24bfe8dcef735cda0e595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681354549176 2023-04-13 02:55:49,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ebafbc65ae544d02af40dfc175735dba 2023-04-13 02:55:49,273 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:49,274 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#472 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:49,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354609274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354609277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354609278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/dac81b65d1654dd885253f23a6299801 2023-04-13 02:55:49,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354609284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354609285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/8514bc7412c2483cb8effb00916feb2e 2023-04-13 02:55:49,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/ebafbc65ae544d02af40dfc175735dba as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba 2023-04-13 02:55:49,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba, entries=300, sequenceid=551, filesize=19.0 K 2023-04-13 02:55:49,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/dac81b65d1654dd885253f23a6299801 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801 2023-04-13 02:55:49,311 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801, entries=150, sequenceid=551, filesize=11.9 K 2023-04-13 02:55:49,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/8514bc7412c2483cb8effb00916feb2e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e 2023-04-13 02:55:49,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,318 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e, entries=150, sequenceid=551, filesize=11.9 K 2023-04-13 02:55:49,320 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,320 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~236.11 KB/241776, currentSize=93.93 KB/96180 for 9a7666d97e523c69aa839344392ace09 in 80ms, sequenceid=551, compaction requested=true 2023-04-13 02:55:49,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,321 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:49,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 02:55:49,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-13 02:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,322 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,322 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:49,322 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,335 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/956e9a19ec574364a7d18017542d2a2c 2023-04-13 02:55:49,350 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/ac4542de0c8e4d99a251782597a4ea16 2023-04-13 02:55:49,361 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/4447208b1822492ab9abb7fc83a5d964 2023-04-13 02:55:49,372 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/956e9a19ec574364a7d18017542d2a2c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c 2023-04-13 02:55:49,380 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c, entries=150, sequenceid=568, filesize=11.9 K 2023-04-13 02:55:49,380 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-13 02:55:49,381 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/ac4542de0c8e4d99a251782597a4ea16 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16 2023-04-13 02:55:49,387 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16, entries=150, sequenceid=568, filesize=11.9 K 2023-04-13 02:55:49,388 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/4447208b1822492ab9abb7fc83a5d964 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964 2023-04-13 02:55:49,393 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964, entries=150, sequenceid=568, filesize=11.9 K 2023-04-13 02:55:49,394 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 9a7666d97e523c69aa839344392ace09 in 72ms, sequenceid=568, compaction requested=true 2023-04-13 02:55:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,394 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,395 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad67792[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,399 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,399 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,401 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,404 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,404 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:49,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:49,406 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:49,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,418 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,418 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,418 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354609444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354549443, max timestamp=1681354549452Num uniques 859; 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354549443, max timestamp=1681354549452Num uniques 859; 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354549443, max timestamp=1681354549452Num uniques 859; 2023-04-13 02:55:49,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:55:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354609490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354609492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354609493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354609497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,518 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,522 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,524 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,526 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,526 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,527 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:49,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749979ed[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,534 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,538 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,538 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,538 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,540 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,545 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,546 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,546 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,546 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354609555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354609600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354609600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354609602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354609602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,646 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,650 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,653 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:49,655 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2b1c38[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,660 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,660 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,662 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,662 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,665 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,665 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,668 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,668 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,670 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,673 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,673 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,673 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,688 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/897ba88e64524e56bc531087e2681823 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/897ba88e64524e56bc531087e2681823 2023-04-13 02:55:49,693 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into 897ba88e64524e56bc531087e2681823(size=13.0 K), total size for store is 36.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:49,693 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,693 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=12, startTime=1681354549240; duration=0sec 2023-04-13 02:55:49,693 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-13 02:55:49,693 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-13 02:55:49,694 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:49,694 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:49,694 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:49,694 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:55:49,696 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/a5ca429757394b0da1c788048093e267 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a5ca429757394b0da1c788048093e267 2023-04-13 02:55:49,697 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:55:49,698 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/C is initiating minor compaction (all files) 2023-04-13 02:55:49,698 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/C in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,698 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e6c33fd81604429782badc6e13ccf34b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=72.3 K 2023-04-13 02:55:49,698 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e6c33fd81604429782badc6e13ccf34b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1681354547581 2023-04-13 02:55:49,699 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6d760376451b4e1aba95da462d5ec21d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1681354547920 2023-04-13 02:55:49,699 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting af1c0392f4154f51a9a7f5aac50247bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1681354548037 2023-04-13 02:55:49,700 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cd88679726f741348001232f465eed3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681354549176 2023-04-13 02:55:49,700 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8514bc7412c2483cb8effb00916feb2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681354549207 2023-04-13 02:55:49,700 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4447208b1822492ab9abb7fc83a5d964, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681354549251 2023-04-13 02:55:49,701 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into a5ca429757394b0da1c788048093e267(size=13.0 K), total size for store is 43.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,701 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=12, startTime=1681354549240; duration=0sec 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 6 compacting, 0 eligible, 16 blocking 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:49,701 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:49,702 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:49,702 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/B is initiating minor compaction (all files) 2023-04-13 02:55:49,702 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/B in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,702 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/897ba88e64524e56bc531087e2681823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=36.8 K 2023-04-13 02:55:49,703 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 897ba88e64524e56bc531087e2681823, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681354549176 2023-04-13 02:55:49,703 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dac81b65d1654dd885253f23a6299801, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681354549207 2023-04-13 02:55:49,704 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ac4542de0c8e4d99a251782597a4ea16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681354549251 2023-04-13 02:55:49,709 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#B#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:49,710 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#C#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:49,728 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/a85b3d197d564bf1a16631f9029a2a32 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a85b3d197d564bf1a16631f9029a2a32 2023-04-13 02:55:49,734 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/484934177f6b4dc5a10a2131f8f3c537 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/484934177f6b4dc5a10a2131f8f3c537 2023-04-13 02:55:49,736 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a7666d97e523c69aa839344392ace09/B of 9a7666d97e523c69aa839344392ace09 into a85b3d197d564bf1a16631f9029a2a32(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:49,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,736 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/B, priority=13, startTime=1681354549321; duration=0sec 2023-04-13 02:55:49,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,739 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9a7666d97e523c69aa839344392ace09/C of 9a7666d97e523c69aa839344392ace09 into 484934177f6b4dc5a10a2131f8f3c537(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:49,739 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,739 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/C, priority=10, startTime=1681354549321; duration=0sec 2023-04-13 02:55:49,739 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354609763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,773 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,776 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,776 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,778 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,781 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,781 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,782 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:49,782 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,787 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724895aa[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,789 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:49,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,794 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,794 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,801 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,802 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354609807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354609807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354609807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354609807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/4800dae64ebe438cac5b6333da5bd506 2023-04-13 02:55:49,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1cd66f3a0ac64c5c9757506ea58b8731 2023-04-13 02:55:49,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e844fb38cfe04c3f8737dbda73270e73 2023-04-13 02:55:49,902 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:49,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:49,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:49,906 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:49,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/4800dae64ebe438cac5b6333da5bd506 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506 2023-04-13 02:55:49,908 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,908 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:49,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:49,910 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,911 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:49,911 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:49,911 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:49,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506, entries=350, sequenceid=579, filesize=21.4 K 2023-04-13 02:55:49,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/1cd66f3a0ac64c5c9757506ea58b8731 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1cd66f3a0ac64c5c9757506ea58b8731 2023-04-13 02:55:49,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,913 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:49,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1cd66f3a0ac64c5c9757506ea58b8731, entries=150, sequenceid=579, filesize=11.9 K 2023-04-13 02:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/e844fb38cfe04c3f8737dbda73270e73 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e844fb38cfe04c3f8737dbda73270e73 2023-04-13 02:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8b0696[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:49,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:49,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,919 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:49,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:49,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:49,926 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:49,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:49,926 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:49,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:49,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:49,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:49,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:49,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:49,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:49,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:49,932 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:49,932 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:49,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:49,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:49,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e844fb38cfe04c3f8737dbda73270e73, entries=150, sequenceid=579, filesize=11.9 K 2023-04-13 02:55:49,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9a7666d97e523c69aa839344392ace09 in 533ms, sequenceid=579, compaction requested=true 2023-04-13 02:55:49,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,938 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:55:49,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,938 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 02:55:49,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:49,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:49,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:49,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:49,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:49,940 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 02:55:49,940 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 66918 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:55:49,940 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 9a7666d97e523c69aa839344392ace09/A is initiating minor compaction (all files) 2023-04-13 02:55:49,940 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a7666d97e523c69aa839344392ace09/A in TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:49,940 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a5ca429757394b0da1c788048093e267, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp, totalSize=65.3 K 2023-04-13 02:55:49,941 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting a5ca429757394b0da1c788048093e267, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1681354549176 2023-04-13 02:55:49,941 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:49,941 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ebafbc65ae544d02af40dfc175735dba, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681354549207 2023-04-13 02:55:49,941 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:49,941 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:49,942 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 956e9a19ec574364a7d18017542d2a2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1681354549251 2023-04-13 02:55:49,942 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4800dae64ebe438cac5b6333da5bd506, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681354549395 2023-04-13 02:55:49,950 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a7666d97e523c69aa839344392ace09#A#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:49,965 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/690c4526b9b64965b8ef9a115ee81574 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/690c4526b9b64965b8ef9a115ee81574 2023-04-13 02:55:49,971 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a7666d97e523c69aa839344392ace09/A of 9a7666d97e523c69aa839344392ace09 into 690c4526b9b64965b8ef9a115ee81574(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:49,971 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:49,971 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09., storeName=9a7666d97e523c69aa839344392ace09/A, priority=12, startTime=1681354549938; duration=0sec 2023-04-13 02:55:49,971 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:50,032 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:50,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:50,036 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:50,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:50,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:50,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:50,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:50,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:50,041 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:50,041 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:50,049 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0f94af0404be479eaf319ba538bed0e5 2023-04-13 02:55:50,058 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/7b942dce1bb7480bbbf4799a9dc96127 2023-04-13 02:55:50,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:50,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:50,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354610116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354610116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354610116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354610123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354610123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354610224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354610231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354610235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354610430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354610440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354610440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,469 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/5688477b8efd49ab9a2f1aeafbebf036 2023-04-13 02:55:50,473 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0f94af0404be479eaf319ba538bed0e5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0f94af0404be479eaf319ba538bed0e5 2023-04-13 02:55:50,476 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0f94af0404be479eaf319ba538bed0e5, entries=150, sequenceid=619, filesize=11.9 K 2023-04-13 02:55:50,477 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/7b942dce1bb7480bbbf4799a9dc96127 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/7b942dce1bb7480bbbf4799a9dc96127 2023-04-13 02:55:50,481 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/7b942dce1bb7480bbbf4799a9dc96127, entries=150, sequenceid=619, filesize=11.9 K 2023-04-13 02:55:50,481 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/5688477b8efd49ab9a2f1aeafbebf036 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/5688477b8efd49ab9a2f1aeafbebf036 2023-04-13 02:55:50,486 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/5688477b8efd49ab9a2f1aeafbebf036, entries=150, sequenceid=619, filesize=11.9 K 2023-04-13 02:55:50,487 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 9a7666d97e523c69aa839344392ace09 in 446ms, sequenceid=619, compaction requested=true 2023-04-13 02:55:50,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:50,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:50,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:50,493 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,493 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e24a1b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:50,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:50,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,496 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:50,497 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:50,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,497 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:50,498 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:50,499 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:50,502 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:50,502 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:50,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:50,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:50,503 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:50,504 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,504 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:50,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,504 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:50,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:50,505 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:50,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:50,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:50,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,512 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:50,512 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:50,512 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:50,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:50,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:50,612 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:50,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:50,615 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:50,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:50,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:50,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:50,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:50,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:50,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,618 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:50,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:50,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:50,619 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:50,619 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. started... 2023-04-13 02:55:50,619 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:50,620 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:50,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:50,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. as already flushing 2023-04-13 02:55:50,628 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0dad422ad64c4b6cb5cbaa443bc773ea 2023-04-13 02:55:50,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354610686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354610693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41016 deadline: 1681354610739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41004 deadline: 1681354610745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40982 deadline: 1681354610746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354610797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354610799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41040 deadline: 1681354611002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41032 deadline: 1681354611002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,041 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/0afe78fa18cc43278c9bf47602a08373 2023-04-13 02:55:51,051 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/72535090c14a425592adab9002fc3060 2023-04-13 02:55:51,056 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/0dad422ad64c4b6cb5cbaa443bc773ea as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0dad422ad64c4b6cb5cbaa443bc773ea 2023-04-13 02:55:51,062 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0dad422ad64c4b6cb5cbaa443bc773ea, entries=150, sequenceid=631, filesize=11.9 K 2023-04-13 02:55:51,063 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/0afe78fa18cc43278c9bf47602a08373 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0afe78fa18cc43278c9bf47602a08373 2023-04-13 02:55:51,066 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0afe78fa18cc43278c9bf47602a08373, entries=150, sequenceid=631, filesize=11.9 K 2023-04-13 02:55:51,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/72535090c14a425592adab9002fc3060 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/72535090c14a425592adab9002fc3060 2023-04-13 02:55:51,070 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/72535090c14a425592adab9002fc3060, entries=150, sequenceid=631, filesize=11.9 K 2023-04-13 02:55:51,070 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9a7666d97e523c69aa839344392ace09 in 451ms, sequenceid=631, compaction requested=true 2023-04-13 02:55:51,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:51,071 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:51,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733f5063[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:51,074 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:51,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,078 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:51,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,079 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:51,080 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:51,083 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:51,083 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:51,083 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:51,083 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:51,083 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:51,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,085 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:51,086 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:51,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:51,087 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:51,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:51,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,090 DEBUG [Thread-2833] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:51,090 DEBUG [Thread-2833] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:51,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:51,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:51,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:51,186 DEBUG [Thread-2838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21359a0e to 127.0.0.1:62948 2023-04-13 02:55:51,187 DEBUG [Thread-2838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,188 DEBUG [Thread-2836] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14f9adf3 to 127.0.0.1:62948 2023-04-13 02:55:51,189 DEBUG [Thread-2836] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,189 DEBUG [Thread-2842] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05a300b1 to 127.0.0.1:62948 2023-04-13 02:55:51,190 DEBUG [Thread-2842] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,190 DEBUG [Thread-2833] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:51,191 DEBUG [Thread-2840] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bbac0b5 to 127.0.0.1:62948 2023-04-13 02:55:51,191 DEBUG [Thread-2840] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,194 DEBUG [Thread-2834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28e3107b to 127.0.0.1:62948 2023-04-13 02:55:51,194 DEBUG [Thread-2834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:51,252 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-13 02:55:51,252 DEBUG [Thread-2823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5cd71f to 127.0.0.1:62948 2023-04-13 02:55:51,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:51,252 DEBUG [Thread-2823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:51,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:51,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:51,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:51,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:51,254 DEBUG [Thread-2831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5de1e3c7 to 127.0.0.1:62948 2023-04-13 02:55:51,254 DEBUG [Thread-2831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,256 DEBUG [Thread-2829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0db388d3 to 127.0.0.1:62948 2023-04-13 02:55:51,256 DEBUG [Thread-2829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,310 DEBUG [Thread-2825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f0735dc to 127.0.0.1:62948 2023-04-13 02:55:51,310 DEBUG [Thread-2825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,311 DEBUG [Thread-2827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14c209bc to 127.0.0.1:62948 2023-04-13 02:55:51,311 DEBUG [Thread-2827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,311 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:55:51,311 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-13 02:55:51,311 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1736 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5208 rows 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1740 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5220 rows 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1755 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5265 rows 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1739 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5216 rows 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(402): scanned 1747 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(403): verified 5241 rows 2023-04-13 02:55:51,312 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:55:51,312 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1df28b68 to 127.0.0.1:62948 2023-04-13 02:55:51,312 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:55:51,314 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:55:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:55:51,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:51,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:51,320 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354551320"}]},"ts":"1681354551320"} 2023-04-13 02:55:51,321 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:55:51,323 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:55:51,323 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, UNASSIGN}] 2023-04-13 02:55:51,325 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, UNASSIGN 2023-04-13 02:55:51,325 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=9a7666d97e523c69aa839344392ace09, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:51,326 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354551325"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354551325"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354551325"}]},"ts":"1681354551325"} 2023-04-13 02:55:51,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:51,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:51,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a7666d97e523c69aa839344392ace09, disabling compactions & flushes 2023-04-13 02:55:51,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:51,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/9456c4e1bd284f5aaeb80285ce9e22f7 2023-04-13 02:55:51,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:52,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/765d3dae42624add9e5a3e3ca6f97187 2023-04-13 02:55:52,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:52,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/90f60c7ae7434350a04ec624667d364b 2023-04-13 02:55:52,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/9456c4e1bd284f5aaeb80285ce9e22f7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9456c4e1bd284f5aaeb80285ce9e22f7 2023-04-13 02:55:52,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9456c4e1bd284f5aaeb80285ce9e22f7, entries=150, sequenceid=652, filesize=11.9 K 2023-04-13 02:55:52,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/765d3dae42624add9e5a3e3ca6f97187 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/765d3dae42624add9e5a3e3ca6f97187 2023-04-13 02:55:52,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/765d3dae42624add9e5a3e3ca6f97187, entries=150, sequenceid=652, filesize=11.9 K 2023-04-13 02:55:52,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/90f60c7ae7434350a04ec624667d364b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/90f60c7ae7434350a04ec624667d364b 2023-04-13 02:55:52,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/90f60c7ae7434350a04ec624667d364b, entries=150, sequenceid=652, filesize=11.9 K 2023-04-13 02:55:52,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for 9a7666d97e523c69aa839344392ace09 in 1240ms, sequenceid=652, compaction requested=true 2023-04-13 02:55:52,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:52,493 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:52,493 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. after waiting 0 ms 2023-04-13 02:55:52,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:52,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:52,493 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9a7666d97e523c69aa839344392ace09 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:55:52,493 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:52,493 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. because compaction request was cancelled 2023-04-13 02:55:52,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=A 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=B 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a7666d97e523c69aa839344392ace09, store=C 2023-04-13 02:55:52,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:52,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/fbf5051f58a44935acc73e63d529cb95 2023-04-13 02:55:53,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/5360bbebbe844f0bbb9e2cdff74a4ade 2023-04-13 02:55:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:53,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/94001b3a0c6042e796c3b33cbc41b780 2023-04-13 02:55:53,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/A/fbf5051f58a44935acc73e63d529cb95 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/fbf5051f58a44935acc73e63d529cb95 2023-04-13 02:55:53,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/fbf5051f58a44935acc73e63d529cb95, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 02:55:53,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/B/5360bbebbe844f0bbb9e2cdff74a4ade as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5360bbebbe844f0bbb9e2cdff74a4ade 2023-04-13 02:55:53,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5360bbebbe844f0bbb9e2cdff74a4ade, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 02:55:53,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/.tmp/C/94001b3a0c6042e796c3b33cbc41b780 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/94001b3a0c6042e796c3b33cbc41b780 2023-04-13 02:55:53,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/94001b3a0c6042e796c3b33cbc41b780, entries=150, sequenceid=659, filesize=11.9 K 2023-04-13 02:55:53,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 9a7666d97e523c69aa839344392ace09 in 1236ms, sequenceid=659, compaction requested=true 2023-04-13 02:55:53,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/abca624643a043d780dd8a94b8a83c67, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/becaaa52da194ac19be960f532653edc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/3b709bee30854310a60bcd8af6090512, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a0ad913376b64fe9b28d6c26d9cfa3d7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/836891b045484748bcda56dfdd804cf6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c696d45a83174a1fae1cc4f9d41034b0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a6ab3e6428b34e989e43ee98a931a1b9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a5ca429757394b0da1c788048093e267, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506] to archive 2023-04-13 02:55:53,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:53,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/abca624643a043d780dd8a94b8a83c67 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/abca624643a043d780dd8a94b8a83c67 2023-04-13 02:55:53,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dc3c78af216c46f9b124fd9045fc26af 2023-04-13 02:55:53,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/800bcdb120c84a6997fdc757feda45c5 2023-04-13 02:55:53,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f0d7746bfa6f4d1c8c4113a95c9d3283 2023-04-13 02:55:53,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1cbce6fb72734b83be90d9f2dd3a8fff 2023-04-13 02:55:53,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0d9362e2fcce4b60bde5d541c8c21121 2023-04-13 02:55:53,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0bd99a0e75064d0c8721149b0727141e 2023-04-13 02:55:53,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a7446702f2b74c03aa1942f1f865aac2 2023-04-13 02:55:53,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/becaaa52da194ac19be960f532653edc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/becaaa52da194ac19be960f532653edc 2023-04-13 02:55:53,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ac4da4288fbc4b71b9ddc9a0fbae3655 2023-04-13 02:55:53,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/df44d2bea8484f8f928cfd7c3fc14981 2023-04-13 02:55:53,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bfb0e62cfe854a8e9fa3d57bb5c52204 2023-04-13 02:55:53,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/3b709bee30854310a60bcd8af6090512 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/3b709bee30854310a60bcd8af6090512 2023-04-13 02:55:53,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/dbfada21032b44e192709885ead364bc 2023-04-13 02:55:53,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a0ad913376b64fe9b28d6c26d9cfa3d7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a0ad913376b64fe9b28d6c26d9cfa3d7 2023-04-13 02:55:53,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f8d567cd45164e338b10d1d71025100f 2023-04-13 02:55:53,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/8d9f1846788e4d6887aab1f56670531b 2023-04-13 02:55:53,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/882dd932c2864e4d9043f9a78646f770 2023-04-13 02:55:53,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/f111dd44968740939cfb755149e07945 2023-04-13 02:55:53,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/bebb4d3916244faebf79cba3aa0db656 2023-04-13 02:55:53,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/341d585cf7834720ac8ef09a7faa118a 2023-04-13 02:55:53,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/304374ba7eb74b6daa7cb1b1e0bf723b 2023-04-13 02:55:53,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/836891b045484748bcda56dfdd804cf6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/836891b045484748bcda56dfdd804cf6 2023-04-13 02:55:53,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/93c040921cb34e31b3b3986c9b0b627c 2023-04-13 02:55:53,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/2ff0013674224c31bce67fc8f1f0f5a8 2023-04-13 02:55:53,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c696d45a83174a1fae1cc4f9d41034b0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c696d45a83174a1fae1cc4f9d41034b0 2023-04-13 02:55:53,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/c3840c9b4a0e4995a12bc7843764e76d 2023-04-13 02:55:53,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/db6ca83148fb4d7d89e0271c264fbd6e 2023-04-13 02:55:53,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a6ab3e6428b34e989e43ee98a931a1b9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a6ab3e6428b34e989e43ee98a931a1b9 2023-04-13 02:55:53,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/27e961eb75b34dbca7d37bcb36cb29fc 2023-04-13 02:55:53,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ce4ebe0ed24c4665b6ab742d7b7d8d8b 2023-04-13 02:55:53,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/1e2721d1a41947ec9f4523660ec8f3b3 2023-04-13 02:55:53,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a5ca429757394b0da1c788048093e267 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/a5ca429757394b0da1c788048093e267 2023-04-13 02:55:53,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/ebafbc65ae544d02af40dfc175735dba 2023-04-13 02:55:53,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/956e9a19ec574364a7d18017542d2a2c 2023-04-13 02:55:53,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/4800dae64ebe438cac5b6333da5bd506 2023-04-13 02:55:53,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1201e5e589644195968b5f7828f06374, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5150cee686f4b20ab63ad06f4d1b352, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/62867cdb2b06452ab89da509b31178de, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/30bfa92b6f9c4920aa12a4944c49c8cf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/6d7977006cd0483ab34c24f0d1fe5189, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5779eab03f04195b7e871d9d5e5a2c5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/c7b444a2efd44112944489926ca06a64, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/897ba88e64524e56bc531087e2681823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16] to archive 2023-04-13 02:55:53,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:53,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1201e5e589644195968b5f7828f06374 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1201e5e589644195968b5f7828f06374 2023-04-13 02:55:53,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/be96534590be43bab7e1bdb07d37d66f 2023-04-13 02:55:53,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/de8a3af6d5cc42e2882463c701f30bd7 2023-04-13 02:55:53,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/f3087dfbd0874bf3ad31ddef77971d93 2023-04-13 02:55:53,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b5bf1e38031d4359b07dd7b5920aaf0a 2023-04-13 02:55:53,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9b16d1e59f26437aa0191f56271aadbb 2023-04-13 02:55:53,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/640f6c0ff4f241f58dd118a12a59deb8 2023-04-13 02:55:53,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5150cee686f4b20ab63ad06f4d1b352 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5150cee686f4b20ab63ad06f4d1b352 2023-04-13 02:55:53,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/eafc2f2f825b401d81740e0a77ac37bc 2023-04-13 02:55:53,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0aeffdc7ca8043898e7de9e17bb6881f 2023-04-13 02:55:53,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/9f2e639da5c74faa8835eb7fa829b47e 2023-04-13 02:55:53,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/62867cdb2b06452ab89da509b31178de to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/62867cdb2b06452ab89da509b31178de 2023-04-13 02:55:53,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3ec576cc36844a7499870ce93e0b0baf 2023-04-13 02:55:53,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1613d0c187e8438ba6b6dffe4556fcdb 2023-04-13 02:55:53,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/30bfa92b6f9c4920aa12a4944c49c8cf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/30bfa92b6f9c4920aa12a4944c49c8cf 2023-04-13 02:55:53,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/3c5a29b21ddb4bdaa93fba68d187249b 2023-04-13 02:55:53,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5d9f8101a69a45b2ad57a07abbcd8fa6 2023-04-13 02:55:53,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/e4884ff94d714019b30fabb64e8997a4 2023-04-13 02:55:53,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/d935c1a385b94f5eb2823c1247b44a0f 2023-04-13 02:55:53,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/2d6840c1ca5c41fc9d4aa777b42b3f1b 2023-04-13 02:55:53,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/69f386c611ec4ab6889aaf322bfb72df 2023-04-13 02:55:53,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/6d7977006cd0483ab34c24f0d1fe5189 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/6d7977006cd0483ab34c24f0d1fe5189 2023-04-13 02:55:53,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/b2af266c9a6e4ec5a44d3f89eee37cd6 2023-04-13 02:55:53,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dd35d95045a641bd83fde81f233f5a8b 2023-04-13 02:55:53,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5779eab03f04195b7e871d9d5e5a2c5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a5779eab03f04195b7e871d9d5e5a2c5 2023-04-13 02:55:53,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/8ae60bbf3da847a9a795f284019d6bbc 2023-04-13 02:55:53,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/cef1f22471be43779d2ddf3c918501ad 2023-04-13 02:55:53,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/c7b444a2efd44112944489926ca06a64 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/c7b444a2efd44112944489926ca06a64 2023-04-13 02:55:53,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4b331629a5c04f1a9290a70d7e43575f 2023-04-13 02:55:53,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/4eede0ec6e884154a674139978b30610 2023-04-13 02:55:53,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/623f8b2148524ecbb3d0042ae54c3562 2023-04-13 02:55:53,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/897ba88e64524e56bc531087e2681823 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/897ba88e64524e56bc531087e2681823 2023-04-13 02:55:53,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ef499e4d8cf24bfe8dcef735cda0e595 2023-04-13 02:55:53,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/dac81b65d1654dd885253f23a6299801 2023-04-13 02:55:53,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/ac4542de0c8e4d99a251782597a4ea16 2023-04-13 02:55:53,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/715f1c30f8564a7ab5230d254d8d5de0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4caf43e521ee4c41a29bb83d83ef5f93, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ef6fb5db00b431199108ca01e8b82ee, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ac0b66e80b7b490a97cdddefb81b2ba0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/d51f078b28084b06a0621263ab8c4abc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e6c33fd81604429782badc6e13ccf34b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964] to archive 2023-04-13 02:55:53,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:55:53,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/715f1c30f8564a7ab5230d254d8d5de0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/715f1c30f8564a7ab5230d254d8d5de0 2023-04-13 02:55:53,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e9d5fb4d7f4d474c8ffa4175e6d2a6bc 2023-04-13 02:55:53,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/1644f119266249f485a3adf1e46fb05f 2023-04-13 02:55:53,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/bf106c628d9447c1acbabf27d587b9b4 2023-04-13 02:55:53,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7b22cc422958487cb734cfcd5b686555 2023-04-13 02:55:53,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ed24aa44b657454fb8bff49f5c50753e 2023-04-13 02:55:53,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/a3765f91ab5244b9a5e1582077b29f00 2023-04-13 02:55:53,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4caf43e521ee4c41a29bb83d83ef5f93 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4caf43e521ee4c41a29bb83d83ef5f93 2023-04-13 02:55:53,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/216d8aa2efbe4a6f80922375dbaaaeb2 2023-04-13 02:55:53,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/f419362c156241e495ed009e573cd9e1 2023-04-13 02:55:53,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/19da798adfa54613b2ab894421e618b1 2023-04-13 02:55:53,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ef6fb5db00b431199108ca01e8b82ee to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ef6fb5db00b431199108ca01e8b82ee 2023-04-13 02:55:53,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/71da31c17de24de9a97418f7397b0486 2023-04-13 02:55:53,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/93f3af59b19a43ea8e38f0c55e392e4f 2023-04-13 02:55:53,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ac0b66e80b7b490a97cdddefb81b2ba0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/ac0b66e80b7b490a97cdddefb81b2ba0 2023-04-13 02:55:53,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8c2ac58d551d4251916082c116ffdc09 2023-04-13 02:55:53,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cfb8002c5cd5453994cbaa6080fba3b8 2023-04-13 02:55:53,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/17ea5121a0814432be232d27bd70a1af 2023-04-13 02:55:53,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0aa01024e4994421a409ab479017754a 2023-04-13 02:55:53,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/32cae96135ad4cf6afd61e0e080e41c7 2023-04-13 02:55:53,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/28fedda3ec7144f88178be81249dd085 2023-04-13 02:55:53,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ab8e2d1e5644cbc84406a218ae82d6f 2023-04-13 02:55:53,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/d51f078b28084b06a0621263ab8c4abc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/d51f078b28084b06a0621263ab8c4abc 2023-04-13 02:55:53,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/398e9e9975ee41cb998a728021a3cfd7 2023-04-13 02:55:53,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/0ed05fd8e13d4f1086e4695da70566a3 2023-04-13 02:55:53,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/dd5b1a8806374449bf547586b260cc28 2023-04-13 02:55:53,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e6c33fd81604429782badc6e13ccf34b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e6c33fd81604429782badc6e13ccf34b 2023-04-13 02:55:53,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/7993535bffa24f878df8375defc40e3b 2023-04-13 02:55:53,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/6d760376451b4e1aba95da462d5ec21d 2023-04-13 02:55:53,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/af1c0392f4154f51a9a7f5aac50247bc 2023-04-13 02:55:53,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/cd88679726f741348001232f465eed3e 2023-04-13 02:55:53,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/8514bc7412c2483cb8effb00916feb2e 2023-04-13 02:55:53,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/4447208b1822492ab9abb7fc83a5d964 2023-04-13 02:55:53,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/recovered.edits/662.seqid, newMaxSeqId=662, maxSeqId=1 2023-04-13 02:55:53,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09. 2023-04-13 02:55:53,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a7666d97e523c69aa839344392ace09: 2023-04-13 02:55:53,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:53,838 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=9a7666d97e523c69aa839344392ace09, regionState=CLOSED 2023-04-13 02:55:53,838 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354553838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354553838"}]},"ts":"1681354553838"} 2023-04-13 02:55:53,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-13 02:55:53,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 9a7666d97e523c69aa839344392ace09, server=jenkins-hbase4.apache.org,34919,1681354416534 in 2.5130 sec 2023-04-13 02:55:53,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-13 02:55:53,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a7666d97e523c69aa839344392ace09, UNASSIGN in 2.5170 sec 2023-04-13 02:55:53,841 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354553841"}]},"ts":"1681354553841"} 2023-04-13 02:55:53,842 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:55:53,843 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:55:53,844 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5300 sec 2023-04-13 02:55:55,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 02:55:55,425 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-13 02:55:55,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:55:55,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,428 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 02:55:55,428 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,433 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:55,435 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/recovered.edits] 2023-04-13 02:55:55,437 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0dad422ad64c4b6cb5cbaa443bc773ea to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0dad422ad64c4b6cb5cbaa443bc773ea 2023-04-13 02:55:55,438 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0f94af0404be479eaf319ba538bed0e5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/0f94af0404be479eaf319ba538bed0e5 2023-04-13 02:55:55,439 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/690c4526b9b64965b8ef9a115ee81574 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/690c4526b9b64965b8ef9a115ee81574 2023-04-13 02:55:55,440 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9456c4e1bd284f5aaeb80285ce9e22f7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/9456c4e1bd284f5aaeb80285ce9e22f7 2023-04-13 02:55:55,443 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/fbf5051f58a44935acc73e63d529cb95 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/A/fbf5051f58a44935acc73e63d529cb95 2023-04-13 02:55:55,445 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0afe78fa18cc43278c9bf47602a08373 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/0afe78fa18cc43278c9bf47602a08373 2023-04-13 02:55:55,446 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1cd66f3a0ac64c5c9757506ea58b8731 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/1cd66f3a0ac64c5c9757506ea58b8731 2023-04-13 02:55:55,447 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5360bbebbe844f0bbb9e2cdff74a4ade to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/5360bbebbe844f0bbb9e2cdff74a4ade 2023-04-13 02:55:55,448 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/765d3dae42624add9e5a3e3ca6f97187 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/765d3dae42624add9e5a3e3ca6f97187 2023-04-13 02:55:55,449 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/7b942dce1bb7480bbbf4799a9dc96127 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/7b942dce1bb7480bbbf4799a9dc96127 2023-04-13 02:55:55,450 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a85b3d197d564bf1a16631f9029a2a32 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/B/a85b3d197d564bf1a16631f9029a2a32 2023-04-13 02:55:55,452 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/484934177f6b4dc5a10a2131f8f3c537 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/484934177f6b4dc5a10a2131f8f3c537 2023-04-13 02:55:55,453 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/5688477b8efd49ab9a2f1aeafbebf036 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/5688477b8efd49ab9a2f1aeafbebf036 2023-04-13 02:55:55,454 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/72535090c14a425592adab9002fc3060 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/72535090c14a425592adab9002fc3060 2023-04-13 02:55:55,455 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/90f60c7ae7434350a04ec624667d364b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/90f60c7ae7434350a04ec624667d364b 2023-04-13 02:55:55,456 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/94001b3a0c6042e796c3b33cbc41b780 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/94001b3a0c6042e796c3b33cbc41b780 2023-04-13 02:55:55,457 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e844fb38cfe04c3f8737dbda73270e73 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/C/e844fb38cfe04c3f8737dbda73270e73 2023-04-13 02:55:55,459 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/recovered.edits/662.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09/recovered.edits/662.seqid 2023-04-13 02:55:55,460 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/9a7666d97e523c69aa839344392ace09 2023-04-13 02:55:55,460 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:55,461 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,463 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:55:55,464 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:55:55,465 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,465 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:55:55,465 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354555465"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:55,466 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:55:55,466 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9a7666d97e523c69aa839344392ace09, NAME => 'TestAcidGuarantees,,1681354529016.9a7666d97e523c69aa839344392ace09.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:55:55,466 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:55:55,466 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354555466"}]},"ts":"9223372036854775807"} 2023-04-13 02:55:55,467 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:55:55,473 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,474 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 47 msec 2023-04-13 02:55:55,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 02:55:55,529 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-13 02:55:55,541 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=623 (was 562) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=470 (was 398) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=1117 (was 783) - AvailableMemoryMB LEAK? - 2023-04-13 02:55:55,553 INFO [Listener at localhost/34379] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=623, MaxFileDescriptor=60000, SystemLoadAverage=470, ProcessCount=173, AvailableMemoryMB=1117 2023-04-13 02:55:55,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:55:55,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:55:55,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:55,557 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 02:55:55,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-13 02:55:55,558 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 02:55:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 02:55:55,559 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:55,560 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 empty. 2023-04-13 02:55:55,560 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:55,560 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:55:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 02:55:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 02:55:55,967 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:55:55,968 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 63bd78eb0ff0948d99f46c700cb54493, NAME => 'TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp 2023-04-13 02:55:55,973 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:55,973 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 63bd78eb0ff0948d99f46c700cb54493, disabling compactions & flushes 2023-04-13 02:55:55,973 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:55,973 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:55,974 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. after waiting 0 ms 2023-04-13 02:55:55,974 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:55,974 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:55,974 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:55,975 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 02:55:55,976 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354555976"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354555976"}]},"ts":"1681354555976"} 2023-04-13 02:55:55,977 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 02:55:55,978 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 02:55:55,978 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354555978"}]},"ts":"1681354555978"} 2023-04-13 02:55:55,979 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 02:55:55,981 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, ASSIGN}] 2023-04-13 02:55:55,983 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, ASSIGN 2023-04-13 02:55:55,983 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=false 2023-04-13 02:55:56,134 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:56,135 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354556134"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354556134"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354556134"}]},"ts":"1681354556134"} 2023-04-13 02:55:56,136 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:56,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 02:55:56,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:56,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 63bd78eb0ff0948d99f46c700cb54493, NAME => 'TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:55:56,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:56,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,292 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,293 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A 2023-04-13 02:55:56,293 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A 2023-04-13 02:55:56,293 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:56,293 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName A 2023-04-13 02:55:56,294 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:56,294 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,295 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B 2023-04-13 02:55:56,295 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B 2023-04-13 02:55:56,295 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:56,295 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName B 2023-04-13 02:55:56,295 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:56,295 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,296 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C 2023-04-13 02:55:56,296 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C 2023-04-13 02:55:56,297 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:56,297 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName C 2023-04-13 02:55:56,297 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:56,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:56,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:55:56,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:56,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 02:55:56,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 63bd78eb0ff0948d99f46c700cb54493; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75118583, jitterRate=0.11935411393642426}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:55:56,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:56,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., pid=56, masterSystemTime=1681354556287 2023-04-13 02:55:56,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:56,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:56,304 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:56,305 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354556304"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354556304"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354556304"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354556304"}]},"ts":"1681354556304"} 2023-04-13 02:55:56,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-13 02:55:56,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 in 169 msec 2023-04-13 02:55:56,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-13 02:55:56,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, ASSIGN in 326 msec 2023-04-13 02:55:56,308 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 02:55:56,309 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354556308"}]},"ts":"1681354556308"} 2023-04-13 02:55:56,309 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 02:55:56,312 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 02:55:56,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 757 msec 2023-04-13 02:55:56,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 02:55:56,663 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-13 02:55:56,663 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x041dc777 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:56,668 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c0766f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:56,669 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:56,671 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:56,672 DEBUG [Listener at localhost/34379] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 02:55:56,674 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 02:55:56,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 02:55:56,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 02:55:56,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 02:55:57,085 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:55:57,086 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 02:55:57,086 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 02:55:57,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 02:55:57,090 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, REOPEN/MOVE}] 2023-04-13 02:55:57,090 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, REOPEN/MOVE 2023-04-13 02:55:57,091 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,091 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354557091"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354557091"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354557091"}]},"ts":"1681354557091"} 2023-04-13 02:55:57,092 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:57,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,244 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 63bd78eb0ff0948d99f46c700cb54493, disabling compactions & flushes 2023-04-13 02:55:57,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. after waiting 0 ms 2023-04-13 02:55:57,245 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 02:55:57,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:57,249 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 63bd78eb0ff0948d99f46c700cb54493 to self. 2023-04-13 02:55:57,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,250 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=CLOSED 2023-04-13 02:55:57,250 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354557250"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354557250"}]},"ts":"1681354557250"} 2023-04-13 02:55:57,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-13 02:55:57,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 in 159 msec 2023-04-13 02:55:57,252 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34919,1681354416534; forceNewPlan=false, retain=true 2023-04-13 02:55:57,403 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,403 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354557403"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354557403"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354557403"}]},"ts":"1681354557403"} 2023-04-13 02:55:57,404 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:55:57,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 63bd78eb0ff0948d99f46c700cb54493, NAME => 'TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.', STARTKEY => '', ENDKEY => ''} 2023-04-13 02:55:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 02:55:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,559 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,560 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,561 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A 2023-04-13 02:55:57,561 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A 2023-04-13 02:55:57,561 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:57,561 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName A 2023-04-13 02:55:57,563 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:57,563 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,563 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B 2023-04-13 02:55:57,564 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B 2023-04-13 02:55:57,564 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:57,564 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName B 2023-04-13 02:55:57,564 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:57,564 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,565 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C 2023-04-13 02:55:57,565 DEBUG [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C 2023-04-13 02:55:57,565 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 02:55:57,565 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63bd78eb0ff0948d99f46c700cb54493 columnFamilyName C 2023-04-13 02:55:57,566 INFO [StoreOpener-63bd78eb0ff0948d99f46c700cb54493-1] regionserver.HStore(310): Store=63bd78eb0ff0948d99f46c700cb54493/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 02:55:57,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 02:55:57,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 63bd78eb0ff0948d99f46c700cb54493; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72622635, jitterRate=0.08216159045696259}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 02:55:57,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:57,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., pid=61, masterSystemTime=1681354557556 2023-04-13 02:55:57,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,572 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,572 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354557572"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681354557572"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681354557572"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681354557572"}]},"ts":"1681354557572"} 2023-04-13 02:55:57,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-13 02:55:57,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 in 169 msec 2023-04-13 02:55:57,575 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-13 02:55:57,575 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, REOPEN/MOVE in 484 msec 2023-04-13 02:55:57,576 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-13 02:55:57,576 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-13 02:55:57,578 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-13 02:55:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-13 02:55:57,579 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x089b6154 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,584 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6257294f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,584 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x01d207bb to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,594 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52104714, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,594 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x5bcc330a to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,598 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a320817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,598 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x23afc0f2 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,602 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19c1e3ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,602 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x0f1adb60 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,607 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@526f2221, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,607 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,611 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,611 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,615 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,616 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x26d7d0b5 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,621 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@156e4ba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,622 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x601410a4 to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,626 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@785e366b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,626 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(139): Connect 0x1f8e4d5e to 127.0.0.1:62948 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 02:55:57,631 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fdfef3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 02:55:57,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:57,644 DEBUG [hconnection-0x319e45c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,645 DEBUG [hconnection-0x33f5ea80-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,646 DEBUG [hconnection-0x7bb208c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,647 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,649 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,652 DEBUG [hconnection-0x2e2b85c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:57,657 DEBUG [hconnection-0x322b7d2c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,657 DEBUG [hconnection-0x2f0f22a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,658 DEBUG [hconnection-0x2b7f3101-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,657 DEBUG [hconnection-0x599f4c08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,657 DEBUG [hconnection-0x772c2fb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,657 DEBUG [hconnection-0x4f55c6c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 02:55:57,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:57,660 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,662 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:57,663 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:57,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:57,663 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,663 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,665 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,665 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,665 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,667 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 02:55:57,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,673 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:57,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:57,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:57,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:57,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:57,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:57,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:57,682 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,682 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:57,684 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:57,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456c7e6b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:57,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:57,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,687 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:57,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:57,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:57,691 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:57,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:57,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:57,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:57,691 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:57,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,692 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,693 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,694 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:57,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:57,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:57,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:57,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:57,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:57,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,696 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,697 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:57,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,697 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:57,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,718 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041368fcaa093ab84f0cbcba13f3ebc76837 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041368fcaa093ab84f0cbcba13f3ebc76837 2023-04-13 02:55:57,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354617710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681354557705, max timestamp=1681354557716Num uniques 859; 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681354557705, max timestamp=1681354557716Num uniques 859; 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681354557708, max timestamp=1681354557716Num uniques 859; 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 02:55:57,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8056, suffix heap size=117064,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=176 2023-04-13 02:55:57,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354617747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354617747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354617748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354617748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,797 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:57,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:57,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:57,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:57,801 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:57,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:57,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:57,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,802 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:57,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:57,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:57,805 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:57,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:57,806 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:57,811 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:57,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d06fe6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:57,813 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,814 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:57,814 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,816 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:57,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:57,818 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:57,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:57,818 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:57,820 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,820 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,820 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,821 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,822 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:57,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:57,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,826 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:57,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,826 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:57,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354617835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354617851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354617851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354617852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:57,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354617852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,926 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:57,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:57,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:57,930 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:57,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:57,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:57,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:57,933 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,933 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:57,934 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:57,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:57,935 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,935 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:57,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301457b5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:57,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:57,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:57,938 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:57,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,938 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:57,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:57,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:57,943 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:57,944 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:57,944 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:57,944 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:57,944 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:57,945 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,945 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,945 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:57,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:57,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:57,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:57,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:57,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:57,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:57,950 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:57,950 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:57,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354618039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,050 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,053 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,053 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354618054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,055 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,055 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354618055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,057 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354618056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354618056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3921b59c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,066 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,068 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,069 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,072 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,073 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:58,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,082 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,082 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,122 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2e11b4b63ac346b6a1f5fd5eb5d85fce 2023-04-13 02:55:58,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/47a08f02d6584e6cbb8dd2a453777abc 2023-04-13 02:55:58,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/2b695009414a46ae89b37f7f2546dd6d 2023-04-13 02:55:58,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2e11b4b63ac346b6a1f5fd5eb5d85fce as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce 2023-04-13 02:55:58,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce, entries=150, sequenceid=16, filesize=25.1 K 2023-04-13 02:55:58,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/47a08f02d6584e6cbb8dd2a453777abc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc 2023-04-13 02:55:58,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc, entries=150, sequenceid=16, filesize=11.6 K 2023-04-13 02:55:58,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/2b695009414a46ae89b37f7f2546dd6d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d 2023-04-13 02:55:58,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d, entries=150, sequenceid=16, filesize=11.6 K 2023-04-13 02:55:58,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=130.47 KB/133602 for 63bd78eb0ff0948d99f46c700cb54493 in 489ms, sequenceid=16, compaction requested=false 2023-04-13 02:55:58,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:58,182 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,186 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,186 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,187 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,188 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,191 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=130.47 KB heapSize=367.18 KB 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:58,191 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:58,192 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:58,192 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:58,224 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041344fad3cb46544cc1a8fe4c7733663135 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344fad3cb46544cc1a8fe4c7733663135 2023-04-13 02:55:58,246 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/c61119f03886486994d8e05a6a1aa270 2023-04-13 02:55:58,274 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/663f96213a36430e8f97e25c0fcef9c1 2023-04-13 02:55:58,298 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=41.02 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/cbfd57be96434b55be7f3b661efbcc3b 2023-04-13 02:55:58,304 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/c61119f03886486994d8e05a6a1aa270 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270 2023-04-13 02:55:58,311 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270, entries=150, sequenceid=51, filesize=25.1 K 2023-04-13 02:55:58,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/663f96213a36430e8f97e25c0fcef9c1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1 2023-04-13 02:55:58,315 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 02:55:58,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/cbfd57be96434b55be7f3b661efbcc3b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b 2023-04-13 02:55:58,324 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 02:55:58,325 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~130.47 KB/133602, heapSize ~367.13 KB/375944, currentSize=0 B/0 for 63bd78eb0ff0948d99f46c700cb54493 in 134ms, sequenceid=51, compaction requested=false 2023-04-13 02:55:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:58,325 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,325 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,327 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f180bf5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,329 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,330 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,331 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,331 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,331 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,336 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,336 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,339 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,339 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,339 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,341 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,341 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,341 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,342 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,343 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,346 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,346 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,346 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:58,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,346 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,347 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,347 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,346 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,347 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,347 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,347 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,347 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,347 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:58,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:58,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:58,401 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041326847f8c0e7342228cdd73aa8beb7e21 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326847f8c0e7342228cdd73aa8beb7e21 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354558408, max timestamp=1681354558414Num uniques 859; 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354558408, max timestamp=1681354558414Num uniques 859; 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354558408, max timestamp=1681354558414Num uniques 859; 2023-04-13 02:55:58,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:58,447 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,450 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,451 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,451 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,453 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,454 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,456 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,456 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354618432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354618433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1681354558429, max timestamp=1681354558437Num uniques 1018; 2023-04-13 02:55:58,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8188, totalHeapSize=18856, min timestamp=1681354558429, max timestamp=1681354558437Num uniques 1018; 2023-04-13 02:55:58,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 02:55:58,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1681354558429, max timestamp=1681354558437Num uniques 1018; 2023-04-13 02:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46612, new segment data size=8188, suffix heap size=133840,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=179 2023-04-13 02:55:58,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 02:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af2e8af[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,461 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,463 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,466 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,466 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,466 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,471 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,475 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,475 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,475 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354618476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354618477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354618480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354618558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354618558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,575 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,579 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354618581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354618581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,583 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,583 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354618583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,587 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4849a5bb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,587 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,590 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,590 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,590 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,591 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,594 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,594 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,595 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,595 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,595 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,596 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,597 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,600 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,600 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,600 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,600 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,700 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,703 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,703 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,703 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,703 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,704 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,704 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,708 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,709 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,710 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,710 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,710 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,710 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,710 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,711 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf0647b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,715 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,715 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,715 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,716 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,719 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,719 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,724 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,724 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,724 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,725 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,726 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,729 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,730 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,729 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,730 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,730 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354618762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354618762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354618783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354618784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354618788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,822 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/275aaea0c7d34c6999e6d910e98f1a32 2023-04-13 02:55:58,830 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,833 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,833 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,834 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,834 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,835 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,835 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,849 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,850 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,850 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,850 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,853 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e89fb3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,855 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,855 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,857 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,857 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,857 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,857 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,857 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,858 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,859 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,861 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,861 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,861 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:58,862 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,862 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,862 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,863 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,867 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,868 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,868 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,868 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,868 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:58,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:58,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:58,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:58,868 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,881 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,881 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:58,881 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:58,981 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:58,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:58,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:58,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:58,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:58,985 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:58,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:58,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:58,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,986 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,987 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:58,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:58,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,987 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:58,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:58,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:58,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:58,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:58,989 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:58,989 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:58,991 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:58,993 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e12acf1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:58,993 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,994 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:58,994 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,994 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:58,997 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:58,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:58,999 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:58,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:58,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:58,999 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:58,999 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,001 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,001 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,003 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,004 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,004 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,005 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,005 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,005 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,005 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:59,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354619066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354619068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354619086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354619089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354619091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,105 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,109 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,112 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:59,113 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,115 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf3a8f5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,118 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,119 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,121 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,123 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,123 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,123 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,125 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,126 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,127 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,132 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,132 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,132 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,133 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,133 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,133 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,133 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/7297d8d252804c42879177de28b9c13f 2023-04-13 02:55:59,233 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,237 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,239 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,239 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:59,256 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,269 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145fca4c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,274 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,274 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,276 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/09e5c2d24a92464ca08f0a62ed945660 2023-04-13 02:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,279 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,279 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/275aaea0c7d34c6999e6d910e98f1a32 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32 2023-04-13 02:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,283 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,285 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32, entries=350, sequenceid=61, filesize=52.3 K 2023-04-13 02:55:59,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/7297d8d252804c42879177de28b9c13f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f 2023-04-13 02:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,293 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,293 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,293 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,293 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,294 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:59,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f, entries=100, sequenceid=61, filesize=9.3 K 2023-04-13 02:55:59,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/09e5c2d24a92464ca08f0a62ed945660 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660 2023-04-13 02:55:59,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660, entries=100, sequenceid=61, filesize=9.3 K 2023-04-13 02:55:59,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=132.44 KB/135616 for 63bd78eb0ff0948d99f46c700cb54493 in 916ms, sequenceid=61, compaction requested=true 2023-04-13 02:55:59,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:59,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:59,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:59,303 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 104998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:59,303 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:59,303 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:55:59,303 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:55:59,303 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,303 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,303 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=102.5 K 2023-04-13 02:55:59,304 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=32.5 K 2023-04-13 02:55:59,304 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2e11b4b63ac346b6a1f5fd5eb5d85fce, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354557675 2023-04-13 02:55:59,304 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 47a08f02d6584e6cbb8dd2a453777abc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354557675 2023-04-13 02:55:59,304 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting c61119f03886486994d8e05a6a1aa270, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354557745 2023-04-13 02:55:59,304 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 663f96213a36430e8f97e25c0fcef9c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354557745 2023-04-13 02:55:59,304 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 275aaea0c7d34c6999e6d910e98f1a32, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354558368 2023-04-13 02:55:59,305 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7297d8d252804c42879177de28b9c13f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354558376 2023-04-13 02:55:59,313 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:59,317 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#506 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:59,341 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/7e18137c4bb44b9681c35934bf2df064 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7e18137c4bb44b9681c35934bf2df064 2023-04-13 02:55:59,347 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 7e18137c4bb44b9681c35934bf2df064(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:59,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,347 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=13, startTime=1681354559302; duration=0sec 2023-04-13 02:55:59,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:59,347 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:59,348 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:59,348 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:55:59,348 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,348 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=32.5 K 2023-04-13 02:55:59,349 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2b695009414a46ae89b37f7f2546dd6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681354557675 2023-04-13 02:55:59,349 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cbfd57be96434b55be7f3b661efbcc3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681354557745 2023-04-13 02:55:59,349 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 09e5c2d24a92464ca08f0a62ed945660, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354558376 2023-04-13 02:55:59,357 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 02:55:59,357 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:59,393 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,399 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/8f68234121654da9b0e1c0d13cc38a3a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/8f68234121654da9b0e1c0d13cc38a3a 2023-04-13 02:55:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,400 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,400 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,408 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 8f68234121654da9b0e1c0d13cc38a3a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:59,408 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,408 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=13, startTime=1681354559303; duration=0sec 2023-04-13 02:55:59,408 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:59,409 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,428 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,428 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=132.44 KB heapSize=375.49 KB 2023-04-13 02:55:59,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:59,428 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:59,429 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:59,429 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:59,429 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:59,429 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:59,477 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137140afbbd0424d7fbdee7e62c08d1b0f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137140afbbd0424d7fbdee7e62c08d1b0f 2023-04-13 02:55:59,495 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/d842513e20d54db29661ce6e392a2ebf 2023-04-13 02:55:59,518 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/22f819c28f6d4caa95cb2da916f9965f 2023-04-13 02:55:59,537 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.98 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/67727d9c63724682b04854628d81c584 2023-04-13 02:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/d842513e20d54db29661ce6e392a2ebf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf 2023-04-13 02:55:59,545 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf, entries=150, sequenceid=116, filesize=25.1 K 2023-04-13 02:55:59,546 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/22f819c28f6d4caa95cb2da916f9965f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f 2023-04-13 02:55:59,549 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f, entries=150, sequenceid=116, filesize=11.6 K 2023-04-13 02:55:59,550 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/67727d9c63724682b04854628d81c584 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584 2023-04-13 02:55:59,554 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584, entries=150, sequenceid=116, filesize=11.6 K 2023-04-13 02:55:59,555 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~132.44 KB/135616, heapSize ~375.45 KB/384456, currentSize=0 B/0 for 63bd78eb0ff0948d99f46c700cb54493 in 127ms, sequenceid=116, compaction requested=false 2023-04-13 02:55:59,555 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,555 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,557 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,557 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73adda73[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,560 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,561 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,567 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,571 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,571 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,571 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,571 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,576 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,576 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,576 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:59,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,581 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,582 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,582 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,582 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,582 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,582 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:59,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354559609, max timestamp=1681354559615Num uniques 859; 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354559609, max timestamp=1681354559615Num uniques 859; 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354559609, max timestamp=1681354559615Num uniques 859; 2023-04-13 02:55:59,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:55:59,631 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136450a0d0bea347eeb1f7c6aac4d92851 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136450a0d0bea347eeb1f7c6aac4d92851 2023-04-13 02:55:59,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354619637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354619637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354619638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354619639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354619641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,649 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/b6f2d1c187ac469db2d16d12723f1ad7 2023-04-13 02:55:59,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/a678a9062a2f4560b3c6a1131d08424b 2023-04-13 02:55:59,682 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,696 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,697 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,697 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,698 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,698 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/5442f28805144b66b1f07b4daa547865 2023-04-13 02:55:59,704 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/b6f2d1c187ac469db2d16d12723f1ad7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7 2023-04-13 02:55:59,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7, entries=350, sequenceid=126, filesize=52.3 K 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:59,712 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/a678a9062a2f4560b3c6a1131d08424b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,712 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,715 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,716 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b, entries=150, sequenceid=126, filesize=11.6 K 2023-04-13 02:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/5442f28805144b66b1f07b4daa547865 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865 2023-04-13 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a395d3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,717 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,718 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,719 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,720 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,720 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,720 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865, entries=150, sequenceid=126, filesize=11.6 K 2023-04-13 02:55:59,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 63bd78eb0ff0948d99f46c700cb54493 in 124ms, sequenceid=126, compaction requested=true 2023-04-13 02:55:59,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,721 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-13 02:55:59,721 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:59,721 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:59,721 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 02:55:59,722 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:55:59,722 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:55:59,722 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. because compaction request was cancelled 2023-04-13 02:55:59,722 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,722 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,723 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:59,723 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:55:59,723 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,723 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7e18137c4bb44b9681c35934bf2df064, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=34.9 K 2023-04-13 02:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,724 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7e18137c4bb44b9681c35934bf2df064, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354557747 2023-04-13 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,725 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 22f819c28f6d4caa95cb2da916f9965f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681354558471 2023-04-13 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,725 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting a678a9062a2f4560b3c6a1131d08424b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354559580 2023-04-13 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,728 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,728 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,728 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,732 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,733 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,733 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#514 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,734 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,734 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,737 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,745 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,745 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,746 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,746 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:55:59,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=147.60 KB heapSize=425.53 KB 2023-04-13 02:55:59,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354559637, max timestamp=1681354559746Num uniques 1018; 2023-04-13 02:55:59,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681354559637, max timestamp=1681354559746Num uniques 1018; 2023-04-13 02:55:59,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:55:59,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:55:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:55:59,757 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/d56750e0889c4bdf83dcf91b0cb66a6c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/d56750e0889c4bdf83dcf91b0cb66a6c 2023-04-13 02:55:59,762 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into d56750e0889c4bdf83dcf91b0cb66a6c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:59,762 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,762 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=13, startTime=1681354559721; duration=0sec 2023-04-13 02:55:59,763 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:55:59,763 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:55:59,764 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:55:59,764 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:55:59,764 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,764 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/8f68234121654da9b0e1c0d13cc38a3a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=34.9 K 2023-04-13 02:55:59,765 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8f68234121654da9b0e1c0d13cc38a3a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354557747 2023-04-13 02:55:59,765 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 67727d9c63724682b04854628d81c584, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681354558471 2023-04-13 02:55:59,765 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5442f28805144b66b1f07b4daa547865, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354559580 2023-04-13 02:55:59,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354619766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354619766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,772 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#516 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:55:59,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354619770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354619771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354619771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,780 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/3ba5588c6d4a4031bb50a3569bb34dd9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/3ba5588c6d4a4031bb50a3569bb34dd9 2023-04-13 02:55:59,784 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 3ba5588c6d4a4031bb50a3569bb34dd9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:59,784 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,784 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=13, startTime=1681354559721; duration=0sec 2023-04-13 02:55:59,784 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:59,846 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,849 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,849 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,850 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,851 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/41012cc60d9642e98f2e0724e93fd0c2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/41012cc60d9642e98f2e0724e93fd0c2 2023-04-13 02:55:59,852 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,852 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,853 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,853 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,853 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,854 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,854 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,854 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:59,854 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,854 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,855 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 41012cc60d9642e98f2e0724e93fd0c2(size=25.2 K), total size for store is 102.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:55:59,855 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:55:59,855 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=13, startTime=1681354559302; duration=0sec 2023-04-13 02:55:59,855 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,856 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,856 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee13032[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,858 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,860 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,860 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,860 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,861 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,864 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,864 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,864 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,865 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,865 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,865 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,866 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,867 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,869 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,869 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,869 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,869 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,869 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,869 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,870 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,870 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,870 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354619872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354619873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354619876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354619877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:55:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354619877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,970 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:55:59,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:55:59,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:55:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:55:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:55:59,974 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:55:59,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:55:59,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:55:59,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,974 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,976 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,976 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,977 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:55:59,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:55:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:55:59,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:55:59,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:55:59,978 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:55:59,978 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,978 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:55:59,978 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:55:59,979 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:55:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:55:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:55:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:55:59,979 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:55:59,981 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,981 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,981 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,981 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:55:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:55:59,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14069808[Count = 0] remaining members to acquire global barrier 2023-04-13 02:55:59,983 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:55:59,983 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,984 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:55:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:55:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:55:59,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,985 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:55:59,986 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:55:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:55:59,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:55:59,989 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:55:59,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:55:59,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:55:59,989 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:55:59,989 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:55:59,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:55:59,992 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:55:59,992 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:55:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:55:59,994 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:55:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:55:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:55:59,997 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,996 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:55:59,997 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:55:59,997 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:55:59,997 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:55:59,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:55:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:55:59,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:55:59,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:55:59,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:55:59,997 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:55:59,997 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:55:59,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:55:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354620076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354620076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354620080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354620080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354620080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,097 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:00,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:00,101 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:00,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:00,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:00,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,101 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,102 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,102 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,103 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,103 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:00,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:00,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:00,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:00,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:00,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:00,105 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:00,107 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:00,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e5b6b9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:00,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:00,109 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:00,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:00,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,112 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,112 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:00,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:00,115 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:00,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:00,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:00,115 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:00,115 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:00,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,116 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,117 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:00,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:00,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,122 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,122 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:00,122 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:00,122 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,123 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041340ddcca4f1924689bf61169b1c611dec to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041340ddcca4f1924689bf61169b1c611dec 2023-04-13 02:56:00,223 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:00,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:00,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:00,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:00,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:00,226 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:00,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:00,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:00,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,226 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,228 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:00,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:00,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:00,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:00,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:00,230 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:00,230 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:00,234 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,234 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:00,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d7cb75b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:00,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:00,236 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,237 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:00,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,237 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:00,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:00,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:00,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:00,241 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:00,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:00,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:00,241 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:00,241 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:00,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,244 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,257 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,260 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,265 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:00,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:00,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,266 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,266 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:00,266 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:00,366 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:00,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:00,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:00,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:00,369 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:00,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:00,370 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:00,371 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,371 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,372 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,372 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,373 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:00,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:00,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:00,374 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:00,374 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:00,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:00,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:00,375 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,376 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:00,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6b6466[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:00,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:00,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:00,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:00,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354620378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354620379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,381 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354620382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:00,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:00,384 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:00,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,384 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:00,384 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:00,384 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:00,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354620383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,384 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:00,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,386 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,388 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354620385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,390 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,390 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:00,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,391 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,391 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:00,391 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,491 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:00,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:00,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:00,495 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:00,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:00,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:00,495 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,503 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,503 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:00,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:00,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:00,505 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,505 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:00,505 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:00,505 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:00,507 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0d67fb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:00,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:00,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:00,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,510 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:00,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:00,514 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:00,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:00,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:00,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:00,515 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:00,522 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,522 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,522 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:00,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:00,523 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:00,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:00,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:00,527 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,528 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:00,528 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:00,527 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:00,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:00,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:00,529 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,573 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/b1190c41cad748038adee6ec139fc2a4 2023-04-13 02:56:00,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/3f2f53d0a5d344c68877935ee6da0e74 2023-04-13 02:56:00,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/b23017dd4576492a9830fda855d3bd36 2023-04-13 02:56:00,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/b1190c41cad748038adee6ec139fc2a4 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4 2023-04-13 02:56:00,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4, entries=150, sequenceid=167, filesize=25.3 K 2023-04-13 02:56:00,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/3f2f53d0a5d344c68877935ee6da0e74 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74 2023-04-13 02:56:00,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74, entries=150, sequenceid=167, filesize=11.8 K 2023-04-13 02:56:00,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/b23017dd4576492a9830fda855d3bd36 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36 2023-04-13 02:56:00,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36, entries=150, sequenceid=167, filesize=11.8 K 2023-04-13 02:56:00,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~84.98 KB/87020, heapSize ~238.52 KB/244240, currentSize=100.63 KB/103050 for 63bd78eb0ff0948d99f46c700cb54493 in 862ms, sequenceid=167, compaction requested=true 2023-04-13 02:56:00,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:00,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:00,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:00,610 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 02:56:00,610 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:00,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:00,610 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:56:00,611 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:56:00,611 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. because compaction request was cancelled 2023-04-13 02:56:00,611 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 02:56:00,611 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 131001 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:00,611 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:00,611 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,611 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/41012cc60d9642e98f2e0724e93fd0c2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=127.9 K 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 41012cc60d9642e98f2e0724e93fd0c2, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681354557747 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. because compaction request was cancelled 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting d842513e20d54db29661ce6e392a2ebf, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681354558471 2023-04-13 02:56:00,612 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b6f2d1c187ac469db2d16d12723f1ad7, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354559580 2023-04-13 02:56:00,613 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting b1190c41cad748038adee6ec139fc2a4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681354559640 2023-04-13 02:56:00,623 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#519 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:00,628 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:00,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:00,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:00,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:00,636 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:00,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:00,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:00,636 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,637 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:00,638 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:00,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:00,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:00,641 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:00,641 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:00,641 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:56:00,641 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:00,641 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:00,642 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:00,642 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:00,642 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:00,642 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:00,642 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/69970bea35cb4886ada5b5df40d98d7c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:56:00,648 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/69970bea35cb4886ada5b5df40d98d7c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/69970bea35cb4886ada5b5df40d98d7c 2023-04-13 02:56:00,655 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 69970bea35cb4886ada5b5df40d98d7c(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:00,655 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:00,655 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=12, startTime=1681354560610; duration=0sec 2023-04-13 02:56:00,655 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:00,661 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413580f58af3a8e4dc6bcf0aadc6ed7ba3d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413580f58af3a8e4dc6bcf0aadc6ed7ba3d 2023-04-13 02:56:00,666 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=188, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ff5dfe91e66a46578b98c7b6df1d0df0 2023-04-13 02:56:00,677 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/40f74d0669874931a583bd6f38cf47b8 2023-04-13 02:56:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:00,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:00,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354620895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354620895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354620898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354620898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:00,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:00,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354620898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354620999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354620999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354621001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354621001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354621001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,086 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/baa36d94c82946c0939cb74f2e2d5993 2023-04-13 02:56:01,090 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ff5dfe91e66a46578b98c7b6df1d0df0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0 2023-04-13 02:56:01,093 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0, entries=150, sequenceid=188, filesize=25.3 K 2023-04-13 02:56:01,095 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/40f74d0669874931a583bd6f38cf47b8 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8 2023-04-13 02:56:01,098 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8, entries=150, sequenceid=188, filesize=11.8 K 2023-04-13 02:56:01,098 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/baa36d94c82946c0939cb74f2e2d5993 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993 2023-04-13 02:56:01,101 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993, entries=150, sequenceid=188, filesize=11.8 K 2023-04-13 02:56:01,102 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for 63bd78eb0ff0948d99f46c700cb54493 in 461ms, sequenceid=188, compaction requested=true 2023-04-13 02:56:01,102 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:01,102 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,104 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,105 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e79407d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,106 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,111 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,111 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,113 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,116 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,116 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,116 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,118 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,119 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,120 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,124 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,124 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,124 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,124 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:01,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:01,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:01,218 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041320c69db268f74effa24e0c73f5752b0e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041320c69db268f74effa24e0c73f5752b0e 2023-04-13 02:56:01,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354621214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354621217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354621218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354621218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354621219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,224 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,224 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=208, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/97f7268f84c24eba8b5bf170ed5bc425 2023-04-13 02:56:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,228 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,228 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,229 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,229 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,230 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:01,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,239 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,239 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,239 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,239 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,241 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c7e486[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,242 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,243 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,246 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,248 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,248 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,248 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,250 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,250 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,250 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,250 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,251 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,254 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:01,254 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,254 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,254 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354621319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354621322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354621324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,354 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,358 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,360 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:01,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,362 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,363 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,365 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273efa6c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,367 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,368 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,369 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,370 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,370 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,372 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,372 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,372 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,372 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,372 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,377 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,380 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,380 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,380 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,380 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,381 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,481 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,484 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,484 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,484 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,484 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,484 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,484 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,485 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,485 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,486 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,487 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,488 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,488 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,489 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d63cf[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,490 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,491 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,492 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,492 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,493 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,496 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,496 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,496 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,505 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,506 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,508 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,511 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,512 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,512 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,512 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354621523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354621526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354621527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354621527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354621527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,612 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,615 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,615 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,616 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,617 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,622 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,624 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,624 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,626 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f3a1df[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/19e95f6f5dff48898ba3dfc6b97e016b 2023-04-13 02:56:01,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,634 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,635 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,635 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,635 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,636 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,650 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,650 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,650 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,651 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,656 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,657 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,657 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:01,757 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,760 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,760 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,761 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,761 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,763 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,763 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:01,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,767 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b3992d3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,770 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,770 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,770 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,772 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,774 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,774 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,774 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,777 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,777 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,777 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,779 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:01,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,782 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,782 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354621827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354621830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354621831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354621831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:01,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354621832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,883 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:01,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:01,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:01,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:01,886 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:01,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:01,886 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:01,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,887 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,891 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,891 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:01,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:01,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:01,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:01,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:01,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:01,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:01,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:01,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:01,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b84e442[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:01,897 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,899 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:01,899 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:01,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,899 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:01,901 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:01,903 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:01,903 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:01,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:01,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:01,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:01,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:01,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:01,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:01,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:01,913 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:01,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:01,914 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:01,914 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:01,913 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:01,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:01,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:01,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:02,014 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:02,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:02,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:02,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:02,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:02,017 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:02,017 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:02,017 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:02,017 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:02,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,019 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:02,019 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:02,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:02,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:02,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:02,021 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,021 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:02,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:02,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:02,023 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:02,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:02,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@365ac7e6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:02,026 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:02,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:02,029 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:02,032 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:02,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:02,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:02,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:02,032 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:02,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,034 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:02,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:02,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:02,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:02,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:02,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:02,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:02,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:02,039 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:02,039 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:02,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:02,039 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/33a494a865154e258f5d8a3101c75b58 2023-04-13 02:56:02,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/97f7268f84c24eba8b5bf170ed5bc425 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425 2023-04-13 02:56:02,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425, entries=200, sequenceid=208, filesize=32.1 K 2023-04-13 02:56:02,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/19e95f6f5dff48898ba3dfc6b97e016b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b 2023-04-13 02:56:02,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b, entries=150, sequenceid=208, filesize=11.8 K 2023-04-13 02:56:02,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/33a494a865154e258f5d8a3101c75b58 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58 2023-04-13 02:56:02,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58, entries=150, sequenceid=208, filesize=11.8 K 2023-04-13 02:56:02,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 63bd78eb0ff0948d99f46c700cb54493 in 889ms, sequenceid=208, compaction requested=true 2023-04-13 02:56:02,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:02,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:02,094 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:02,094 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:02,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:02,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:02,095 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:02,095 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:02,095 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,095 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/69970bea35cb4886ada5b5df40d98d7c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=82.9 K 2023-04-13 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:02,096 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,096 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/d56750e0889c4bdf83dcf91b0cb66a6c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=47.1 K 2023-04-13 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 69970bea35cb4886ada5b5df40d98d7c, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681354559640 2023-04-13 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting d56750e0889c4bdf83dcf91b0cb66a6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354559580 2023-04-13 02:56:02,096 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ff5dfe91e66a46578b98c7b6df1d0df0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681354559764 2023-04-13 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3f2f53d0a5d344c68877935ee6da0e74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681354559640 2023-04-13 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 97f7268f84c24eba8b5bf170ed5bc425, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:02,097 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 40f74d0669874931a583bd6f38cf47b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681354559764 2023-04-13 02:56:02,098 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 19e95f6f5dff48898ba3dfc6b97e016b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:02,105 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:02,106 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#526 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:02,139 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:02,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:02,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:02,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:02,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:02,143 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:02,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:02,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:02,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:02,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,144 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,144 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:02,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,145 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:02,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:02,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:02,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:02,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:02,147 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:02,147 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:02,166 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413af150b67bdd64b1080f691fbf9bbffd3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413af150b67bdd64b1080f691fbf9bbffd3 2023-04-13 02:56:02,172 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ade9a8efdbbd4514ad876f26084d558b 2023-04-13 02:56:02,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:02,333 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:02,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354622343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354622344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354622349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354622348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354622348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354622449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354622450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354622454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354622454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354622454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,518 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/9cb763b352e140cb9f320ea37c31136f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9cb763b352e140cb9f320ea37c31136f 2023-04-13 02:56:02,523 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 9cb763b352e140cb9f320ea37c31136f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:02,523 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:02,523 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=12, startTime=1681354562094; duration=0sec 2023-04-13 02:56:02,524 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:02,524 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:02,525 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:02,525 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:02,525 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:02,525 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/3ba5588c6d4a4031bb50a3569bb34dd9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=47.1 K 2023-04-13 02:56:02,526 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3ba5588c6d4a4031bb50a3569bb34dd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1681354559580 2023-04-13 02:56:02,526 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b23017dd4576492a9830fda855d3bd36, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681354559640 2023-04-13 02:56:02,526 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting baa36d94c82946c0939cb74f2e2d5993, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1681354559764 2023-04-13 02:56:02,527 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 33a494a865154e258f5d8a3101c75b58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:02,534 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:02,534 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/9d4a1cce24bb439c9f77da561ab1bf19 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9d4a1cce24bb439c9f77da561ab1bf19 2023-04-13 02:56:02,541 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 9d4a1cce24bb439c9f77da561ab1bf19(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:02,541 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:02,541 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=13, startTime=1681354562094; duration=0sec 2023-04-13 02:56:02,541 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:02,546 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd 2023-04-13 02:56:02,550 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 31f98ef8b2e34b00a3fa0c90c0fdb4bd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:02,550 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:02,550 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=12, startTime=1681354562094; duration=0sec 2023-04-13 02:56:02,550 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:02,598 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/9c86830e91ac423c97acd0bf8f21487d 2023-04-13 02:56:02,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354622651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354622653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354622656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354622656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354622657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354622956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354622958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354622959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354622959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:02,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:02,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354622961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,010 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/c7c22131eddb4e28bc7b9f2f303d8ed6 2023-04-13 02:56:03,014 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ade9a8efdbbd4514ad876f26084d558b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b 2023-04-13 02:56:03,017 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b, entries=150, sequenceid=223, filesize=25.3 K 2023-04-13 02:56:03,018 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/9c86830e91ac423c97acd0bf8f21487d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d 2023-04-13 02:56:03,021 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d, entries=150, sequenceid=223, filesize=11.8 K 2023-04-13 02:56:03,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/c7c22131eddb4e28bc7b9f2f303d8ed6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6 2023-04-13 02:56:03,025 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6, entries=150, sequenceid=223, filesize=11.8 K 2023-04-13 02:56:03,025 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 63bd78eb0ff0948d99f46c700cb54493 in 878ms, sequenceid=223, compaction requested=false 2023-04-13 02:56:03,026 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:03,026 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:03,026 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:03,028 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@91875ab[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:03,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:03,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,031 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:03,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:03,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:03,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:03,035 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:03,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:03,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:03,035 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:03,035 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:03,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:03,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:03,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:03,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:03,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:03,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:03,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:03,042 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:03,042 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:03,042 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:03,142 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:03,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:03,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:03,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:03,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:03,152 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:03,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:03,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:03,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:03,152 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:03,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:03,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:03,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:03,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:03,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:03,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:03,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:03,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:03,156 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:03,159 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:03,159 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:03,160 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 02:56:03,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:03,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:03,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:03,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:03,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:03,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:03,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:03,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:03,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354623470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354623470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354623470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354623472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354623473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354623574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354623574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354623574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354623576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354623576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,595 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041384b820e2b7dd49bd9d9065bfa5c0bf0d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041384b820e2b7dd49bd9d9065bfa5c0bf0d 2023-04-13 02:56:03,599 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/45f55e87be094ad794010004d6f39982 2023-04-13 02:56:03,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354623776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354623777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354623777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354623780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:03,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354623780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,007 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/5198af817b2840ad83fe8e98a2e81abf 2023-04-13 02:56:04,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354624080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354624080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354624081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354624084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354624085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,428 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/321fa12a698847fa85260cc090d5e607 2023-04-13 02:56:04,433 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/45f55e87be094ad794010004d6f39982 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982 2023-04-13 02:56:04,436 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982, entries=150, sequenceid=244, filesize=25.3 K 2023-04-13 02:56:04,437 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/5198af817b2840ad83fe8e98a2e81abf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf 2023-04-13 02:56:04,441 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf, entries=150, sequenceid=244, filesize=11.8 K 2023-04-13 02:56:04,442 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/321fa12a698847fa85260cc090d5e607 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607 2023-04-13 02:56:04,445 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607, entries=150, sequenceid=244, filesize=11.8 K 2023-04-13 02:56:04,446 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 63bd78eb0ff0948d99f46c700cb54493 in 1286ms, sequenceid=244, compaction requested=true 2023-04-13 02:56:04,446 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:04,446 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:04,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:04,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:04,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e70d151[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:04,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:04,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,452 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:04,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:04,452 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:04,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:04,458 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:04,458 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:04,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:04,460 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:04,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:04,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:04,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:04,460 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:04,462 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,462 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:04,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,463 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:04,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,463 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,465 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,469 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:04,469 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:04,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:04,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:04,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:04,469 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:04,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:04,470 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:04,469 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:04,470 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:04,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:04,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:04,470 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:04,570 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:04,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:04,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:04,573 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:04,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:04,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:04,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:04,574 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,574 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:04,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,576 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:04,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:04,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,577 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:04,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:04,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:04,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:04,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:04,578 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:04,578 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:04,578 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:04,578 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:04,579 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:04,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:04,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:04,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354624595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354624595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354624596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354624598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354624599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,602 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c643d0c22c0d4f3db3b4d0f4bab9ae5b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c643d0c22c0d4f3db3b4d0f4bab9ae5b 2023-04-13 02:56:04,607 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/161ba1be88a04f54bb6edd27b8c72042 2023-04-13 02:56:04,618 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/b80634f3280f45d08e64b27456ad0532 2023-04-13 02:56:04,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354624700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354624700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354624700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354624700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354624702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354624903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354624904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354624904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354624904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:04,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354624904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,034 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/b5a1ce69785e4488bc3f29f6c3922029 2023-04-13 02:56:05,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/161ba1be88a04f54bb6edd27b8c72042 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042 2023-04-13 02:56:05,041 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042, entries=150, sequenceid=259, filesize=25.4 K 2023-04-13 02:56:05,042 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/b80634f3280f45d08e64b27456ad0532 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532 2023-04-13 02:56:05,046 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532, entries=150, sequenceid=259, filesize=11.9 K 2023-04-13 02:56:05,047 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/b5a1ce69785e4488bc3f29f6c3922029 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029 2023-04-13 02:56:05,050 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029, entries=150, sequenceid=259, filesize=11.9 K 2023-04-13 02:56:05,050 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 63bd78eb0ff0948d99f46c700cb54493 in 472ms, sequenceid=259, compaction requested=true 2023-04-13 02:56:05,050 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:05,051 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32aa9a29[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,060 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,060 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,060 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:05,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,068 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,068 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,068 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,168 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,173 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,173 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,175 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,175 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,177 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,189 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:05,189 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:05,190 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,209 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f430dcdf4bf3462f89ad1bcfb54f751c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f430dcdf4bf3462f89ad1bcfb54f751c 2023-04-13 02:56:05,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:05,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,214 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/8d2b467769d446b0ba6f8ee7082f765c 2023-04-13 02:56:05,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354625218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354625218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354625220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,223 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/560286be63544b4e95f6e5e90250d3ce 2023-04-13 02:56:05,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354625223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354625223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,233 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/c4383c271deb42a391a6e60c303520b5 2023-04-13 02:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/8d2b467769d446b0ba6f8ee7082f765c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c 2023-04-13 02:56:05,240 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c, entries=150, sequenceid=276, filesize=25.4 K 2023-04-13 02:56:05,242 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/560286be63544b4e95f6e5e90250d3ce as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce 2023-04-13 02:56:05,245 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce, entries=150, sequenceid=276, filesize=11.9 K 2023-04-13 02:56:05,245 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/c4383c271deb42a391a6e60c303520b5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5 2023-04-13 02:56:05,248 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5, entries=150, sequenceid=276, filesize=11.9 K 2023-04-13 02:56:05,249 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 63bd78eb0ff0948d99f46c700cb54493 in 60ms, sequenceid=276, compaction requested=true 2023-04-13 02:56:05,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:05,249 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,249 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,251 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,252 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,252 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af1e6d[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,253 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,253 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,259 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,259 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,261 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,261 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,263 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,263 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,265 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,266 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:05,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,271 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,271 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:05,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:05,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:05,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354625337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354625337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354625340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,348 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413781dd5cd1fa241d799b7baef8845721c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413781dd5cd1fa241d799b7baef8845721c 2023-04-13 02:56:05,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354625345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354625348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,371 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,375 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,376 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,378 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,378 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,380 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,386 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,388 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,388 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,389 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d09cbe[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,390 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,391 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,392 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,393 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,393 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,394 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,398 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,398 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,398 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,402 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,402 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,402 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,404 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,406 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,409 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,409 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,410 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,410 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,410 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,410 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354625447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354625449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354625449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354625453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354625453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,511 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,514 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,516 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,517 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,521 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,521 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,528 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,528 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,533 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,533 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,534 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50623cac[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,537 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,538 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,541 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,541 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,544 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,548 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,549 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,549 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,549 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:05,649 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,653 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,653 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354625651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354625652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,655 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,656 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354625654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,659 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ff9f5b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,661 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354625658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354625658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,668 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,669 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,670 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,675 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,676 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,676 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:05,763 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/50e4139d2bec4305b8c6d8b3a1567867 2023-04-13 02:56:05,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/93b4b8d816484496b8d49f1b5c451d9d 2023-04-13 02:56:05,776 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,780 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,780 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,782 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,784 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,784 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,787 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c09a39f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,789 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,790 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,791 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,793 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,793 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,793 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,795 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,796 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,801 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,801 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,901 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:05,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:05,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:05,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:05,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:05,904 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:05,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:05,904 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:05,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,906 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:05,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:05,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:05,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:05,908 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:05,910 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:05,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0f9cc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:05,911 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:05,912 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,916 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,916 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:05,916 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:05,916 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:05,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:05,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:05,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:05,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:05,920 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:05,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:05,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:05,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:05,920 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:05,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,921 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:05,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:05,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,923 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:05,926 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:05,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:05,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:05,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,926 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:05,926 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:05,926 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:05,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:05,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37692 deadline: 1681354625957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354625957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354625959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354625963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:05,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:05,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354625964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,026 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:06,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:06,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:06,029 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:06,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:06,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:06,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:06,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:06,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:06,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:06,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:06,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:06,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:06,034 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:06,035 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,036 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,036 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,036 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:06,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41663228[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:06,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,038 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:06,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:06,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:06,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:06,040 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:06,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:06,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:06,042 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:06,042 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:06,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:06,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:06,043 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:06,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,045 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:06,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,046 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:06,048 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:06,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,049 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:06,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:06,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:06,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:06,049 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:06,049 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:06,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:06,149 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:06,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:06,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:06,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:06,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:06,153 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:06,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:06,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:06,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,154 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,155 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:06,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:06,155 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:06,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:06,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:06,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:06,157 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:06,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:06,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:06,157 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:06,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:06,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:06,158 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,159 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:06,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b296818[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:06,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:06,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,163 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:06,163 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:06,163 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:06,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,164 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:06,165 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,165 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:06,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:06,168 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:06,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:06,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:06,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:06,168 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:06,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:06,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,170 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:06,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:06,170 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:06,172 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:06,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:06,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:06,174 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:06,174 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:06,174 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:06,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:06,174 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:06,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:06,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:06,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/0d7cf1cc87884fc79d92ca4a0a58adaa 2023-04-13 02:56:06,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/50e4139d2bec4305b8c6d8b3a1567867 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867 2023-04-13 02:56:06,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867, entries=250, sequenceid=293, filesize=39.2 K 2023-04-13 02:56:06,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/93b4b8d816484496b8d49f1b5c451d9d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d 2023-04-13 02:56:06,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d, entries=150, sequenceid=293, filesize=11.9 K 2023-04-13 02:56:06,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/0d7cf1cc87884fc79d92ca4a0a58adaa as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa 2023-04-13 02:56:06,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa, entries=150, sequenceid=293, filesize=11.9 K 2023-04-13 02:56:06,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 63bd78eb0ff0948d99f46c700cb54493 in 870ms, sequenceid=293, compaction requested=true 2023-04-13 02:56:06,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:06,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:06,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:06,196 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:56:06,196 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:56:06,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:06,198 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170202 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:56:06,198 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72958 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:56:06,198 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:06,198 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:06,198 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,198 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,198 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9d4a1cce24bb439c9f77da561ab1bf19, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=166.2 K 2023-04-13 02:56:06,198 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9cb763b352e140cb9f320ea37c31136f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=71.2 K 2023-04-13 02:56:06,199 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9d4a1cce24bb439c9f77da561ab1bf19, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:06,199 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9cb763b352e140cb9f320ea37c31136f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:06,199 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ade9a8efdbbd4514ad876f26084d558b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681354561215 2023-04-13 02:56:06,200 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 9c86830e91ac423c97acd0bf8f21487d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681354561215 2023-04-13 02:56:06,200 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 45f55e87be094ad794010004d6f39982, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681354562340 2023-04-13 02:56:06,200 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5198af817b2840ad83fe8e98a2e81abf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681354562340 2023-04-13 02:56:06,200 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 161ba1be88a04f54bb6edd27b8c72042, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681354563468 2023-04-13 02:56:06,201 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b80634f3280f45d08e64b27456ad0532, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681354563468 2023-04-13 02:56:06,201 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 8d2b467769d446b0ba6f8ee7082f765c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681354564591 2023-04-13 02:56:06,201 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 560286be63544b4e95f6e5e90250d3ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681354564591 2023-04-13 02:56:06,201 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 50e4139d2bec4305b8c6d8b3a1567867, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565214 2023-04-13 02:56:06,202 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 93b4b8d816484496b8d49f1b5c451d9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565220 2023-04-13 02:56:06,212 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#545 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:06,215 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#544 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:06,234 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/7905abb9df384eb2ba3f39c91b43a12a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7905abb9df384eb2ba3f39c91b43a12a 2023-04-13 02:56:06,238 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 7905abb9df384eb2ba3f39c91b43a12a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:06,238 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:06,238 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=10, startTime=1681354566196; duration=0sec 2023-04-13 02:56:06,238 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:06,238 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 02:56:06,239 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72958 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 02:56:06,239 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:06,239 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,240 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=71.2 K 2023-04-13 02:56:06,240 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 31f98ef8b2e34b00a3fa0c90c0fdb4bd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681354560897 2023-04-13 02:56:06,240 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c7c22131eddb4e28bc7b9f2f303d8ed6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681354561215 2023-04-13 02:56:06,241 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 321fa12a698847fa85260cc090d5e607, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681354562340 2023-04-13 02:56:06,241 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting b5a1ce69785e4488bc3f29f6c3922029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681354563468 2023-04-13 02:56:06,241 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c4383c271deb42a391a6e60c303520b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681354564591 2023-04-13 02:56:06,242 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0d7cf1cc87884fc79d92ca4a0a58adaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565220 2023-04-13 02:56:06,251 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#546 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:06,263 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/0f4af183235943be8827d9f90e85915e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0f4af183235943be8827d9f90e85915e 2023-04-13 02:56:06,267 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 0f4af183235943be8827d9f90e85915e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:06,267 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:06,268 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=10, startTime=1681354566196; duration=0sec 2023-04-13 02:56:06,268 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:06,274 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:06,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:06,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:06,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:06,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:06,278 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:06,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:06,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:06,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:06,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:06,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,282 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:06,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:06,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:06,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:06,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:06,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:06,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:06,284 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:06,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:06,284 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:06,285 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:06,285 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:06,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:06,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:06,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354626475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354626477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354626478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354626478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354626479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354626579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354626582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37700 deadline: 1681354626582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354626582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354626582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,703 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413519de5d032934622aa6f4d6d6457c337 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413519de5d032934622aa6f4d6d6457c337 2023-04-13 02:56:06,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354626785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37678 deadline: 1681354626785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354626786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354626787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:06,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354626787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,044 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/5ef0ccf61ad04a399de4b21351a44d56 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/5ef0ccf61ad04a399de4b21351a44d56 2023-04-13 02:56:07,049 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 5ef0ccf61ad04a399de4b21351a44d56(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:07,049 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:07,049 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=10, startTime=1681354566196; duration=0sec 2023-04-13 02:56:07,049 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:07,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37714 deadline: 1681354627091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354627091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37702 deadline: 1681354627091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354627092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354627092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,108 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/3c4fa7c2a4264361a6524c093bea1884 2023-04-13 02:56:07,518 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/1e1d3bafb24549d689c4afdcffe658b5 2023-04-13 02:56:07,527 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/098987fb027c455cba612ab918eefdab 2023-04-13 02:56:07,531 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/3c4fa7c2a4264361a6524c093bea1884 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884 2023-04-13 02:56:07,535 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884, entries=150, sequenceid=311, filesize=25.4 K 2023-04-13 02:56:07,535 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/1e1d3bafb24549d689c4afdcffe658b5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5 2023-04-13 02:56:07,538 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5, entries=150, sequenceid=311, filesize=11.9 K 2023-04-13 02:56:07,539 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/098987fb027c455cba612ab918eefdab as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab 2023-04-13 02:56:07,541 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab, entries=150, sequenceid=311, filesize=11.9 K 2023-04-13 02:56:07,542 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 63bd78eb0ff0948d99f46c700cb54493 in 1258ms, sequenceid=311, compaction requested=false 2023-04-13 02:56:07,542 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:07,542 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:07,542 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,544 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:07,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6d37b9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:07,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:07,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,549 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,550 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,551 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,551 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:07,553 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:07,553 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:07,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:07,554 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:07,554 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:07,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,559 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,560 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,562 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,564 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,565 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,565 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:07,565 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:07,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:07,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:07,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354627603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354627604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354627605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354627605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354627607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,611 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133bca2b07328b447aababf8cbb886eb98 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133bca2b07328b447aababf8cbb886eb98 2023-04-13 02:56:07,665 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:07,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:07,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:07,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:07,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:07,669 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:07,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:07,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:07,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,672 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:07,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:07,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:07,673 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:07,673 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:07,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:07,674 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:07,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:07,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:07,674 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:07,675 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,675 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:07,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d84dba1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:07,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:07,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,678 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:07,679 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:07,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:07,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:07,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:07,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:07,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:07,683 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:07,683 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,685 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,686 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,688 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,690 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,691 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:07,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:07,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:07,691 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:07,691 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:07,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354627707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354627708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354627710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354627712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354627715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,791 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:07,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:07,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:07,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:07,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:07,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:07,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:07,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:07,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,797 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:07,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:07,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:07,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:07,798 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:07,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:07,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:07,799 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:07,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,802 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:07,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d83554b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:07,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:07,804 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:07,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:07,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,807 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:07,808 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:07,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:07,811 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:07,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:07,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:07,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:07,811 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:07,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,813 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,814 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,816 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:07,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:07,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,818 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,818 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,818 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:07,819 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:07,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354627911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354627912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354627913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354627914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:07,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354627916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,919 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:07,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:07,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:07,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:07,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:07,922 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:07,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:07,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:07,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,925 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:07,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:07,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:07,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:07,927 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:07,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,930 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2a814b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:07,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:07,931 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:07,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:07,932 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:07,936 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:07,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:07,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:07,937 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:07,937 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:07,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,939 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,940 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:07,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:07,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:07,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:07,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:07,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,947 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:07,947 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:07,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:07,947 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:07,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,016 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/822009fbbaec4b2a883349975dfba7d5 2023-04-13 02:56:08,047 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:08,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:08,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:08,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:08,050 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:08,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:08,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:08,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:08,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:08,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:08,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:08,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,053 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:08,054 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,054 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:08,054 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:08,054 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:08,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:08,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:08,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e49a3a8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:08,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:08,059 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,061 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:08,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,063 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:08,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:08,069 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:08,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:08,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:08,069 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:08,069 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:08,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,071 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:08,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:08,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,076 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:08,076 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:08,075 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:08,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,176 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:08,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:08,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:08,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:08,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:08,180 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,182 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,182 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:08,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,183 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:08,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:08,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:08,184 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:08,184 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,186 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256b3213[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:08,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:08,189 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,190 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:08,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:08,190 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:08,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:08,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:08,195 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:08,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:08,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:08,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:08,195 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:08,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,199 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:08,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:08,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,203 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:08,203 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354628217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354628217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354628217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354628218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354628220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,303 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:08,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:08,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:08,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:08,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:08,307 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:08,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:08,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:08,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,310 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:08,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:08,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:08,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:08,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:08,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:08,314 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,314 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,315 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:08,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f502f8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:08,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:08,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:08,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:08,318 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:08,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,319 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:08,322 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:08,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:08,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:08,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:08,322 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:08,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,323 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,326 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:08,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:08,331 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,331 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,332 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:08,332 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:08,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,432 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:08,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:08,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:08,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:08,436 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,436 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,438 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:08,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:08,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:08,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:08,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:08,440 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:08,442 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:08,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2becf934[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:08,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:08,444 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,445 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:08,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,446 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:08,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/6ff591c97dc54b119d7a9fe3aa18e807 2023-04-13 02:56:08,448 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,448 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:08,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:08,453 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:08,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:08,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:08,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:08,453 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:08,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,454 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,455 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:08,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:08,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/55d7be54fc9544e89c4dafa53d812c18 2023-04-13 02:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:08,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:08,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:08,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,461 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:08,461 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:08,461 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:08,461 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:08,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:08,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/822009fbbaec4b2a883349975dfba7d5 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5 2023-04-13 02:56:08,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5, entries=150, sequenceid=332, filesize=25.4 K 2023-04-13 02:56:08,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/6ff591c97dc54b119d7a9fe3aa18e807 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807 2023-04-13 02:56:08,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807, entries=150, sequenceid=332, filesize=11.9 K 2023-04-13 02:56:08,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/55d7be54fc9544e89c4dafa53d812c18 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18 2023-04-13 02:56:08,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18, entries=150, sequenceid=332, filesize=11.9 K 2023-04-13 02:56:08,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 63bd78eb0ff0948d99f46c700cb54493 in 879ms, sequenceid=332, compaction requested=true 2023-04-13 02:56:08,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:08,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:08,477 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:08,478 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:08,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:08,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:08,478 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78684 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:08,478 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:08,478 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,479 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/5ef0ccf61ad04a399de4b21351a44d56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=76.8 K 2023-04-13 02:56:08,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:08,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:08,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7905abb9df384eb2ba3f39c91b43a12a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.2 K 2023-04-13 02:56:08,479 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 5ef0ccf61ad04a399de4b21351a44d56, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565220 2023-04-13 02:56:08,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7905abb9df384eb2ba3f39c91b43a12a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565220 2023-04-13 02:56:08,480 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 3c4fa7c2a4264361a6524c093bea1884, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681354565336 2023-04-13 02:56:08,480 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1e1d3bafb24549d689c4afdcffe658b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681354565336 2023-04-13 02:56:08,480 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 822009fbbaec4b2a883349975dfba7d5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:08,480 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 6ff591c97dc54b119d7a9fe3aa18e807, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:08,488 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#554 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:08,493 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#553 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:08,561 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:08,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:08,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:08,576 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:08,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:08,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:08,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,576 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,578 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:08,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,578 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:08,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:08,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:08,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:08,579 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:08,580 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:08,580 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 02:56:08,580 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:08,581 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:08,581 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:08,581 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:08,581 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:08,581 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:08,597 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304130b83b8545bb74accb87e5714b23b1f12 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130b83b8545bb74accb87e5714b23b1f12 2023-04-13 02:56:08,605 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=344, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/07de2cba946e4f30914981d377a08b02 2023-04-13 02:56:08,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:08,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:08,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354628733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354568731, max timestamp=1681354568735Num uniques 859; 2023-04-13 02:56:08,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354568731, max timestamp=1681354568735Num uniques 859; 2023-04-13 02:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354568731, max timestamp=1681354568735Num uniques 859; 2023-04-13 02:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:08,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354628748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354628749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354628750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354628750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354628838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354628854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354628854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354628854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354628854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:08,900 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/33b4ca87a9e14cc5b505848505114303 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/33b4ca87a9e14cc5b505848505114303 2023-04-13 02:56:08,904 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 33b4ca87a9e14cc5b505848505114303(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:08,904 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:08,904 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=13, startTime=1681354568477; duration=0sec 2023-04-13 02:56:08,904 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:08,905 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:08,909 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:08,910 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:08,910 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:08,910 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0f4af183235943be8827d9f90e85915e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.2 K 2023-04-13 02:56:08,911 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0f4af183235943be8827d9f90e85915e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681354565220 2023-04-13 02:56:08,912 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 098987fb027c455cba612ab918eefdab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1681354565336 2023-04-13 02:56:08,914 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 55d7be54fc9544e89c4dafa53d812c18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:08,925 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/7d36889bc6194b63987538241649524d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/7d36889bc6194b63987538241649524d 2023-04-13 02:56:08,931 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 7d36889bc6194b63987538241649524d(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:08,931 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:08,931 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=13, startTime=1681354568477; duration=0sec 2023-04-13 02:56:08,931 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:08,945 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#557 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:08,954 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/c37396194694491a9223c26015f4adbc as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c37396194694491a9223c26015f4adbc 2023-04-13 02:56:08,957 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into c37396194694491a9223c26015f4adbc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:08,957 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:08,957 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=13, startTime=1681354568478; duration=0sec 2023-04-13 02:56:08,957 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:09,016 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/96400567d3cf41da836555f5eea00d3c 2023-04-13 02:56:09,025 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/5cd089e85d684c118ebcaf312810e823 2023-04-13 02:56:09,030 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/07de2cba946e4f30914981d377a08b02 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02 2023-04-13 02:56:09,033 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02, entries=150, sequenceid=344, filesize=25.4 K 2023-04-13 02:56:09,034 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/96400567d3cf41da836555f5eea00d3c as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c 2023-04-13 02:56:09,038 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c, entries=150, sequenceid=344, filesize=11.9 K 2023-04-13 02:56:09,039 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/5cd089e85d684c118ebcaf312810e823 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823 2023-04-13 02:56:09,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354629045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,048 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823, entries=150, sequenceid=344, filesize=11.9 K 2023-04-13 02:56:09,049 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 63bd78eb0ff0948d99f46c700cb54493 in 469ms, sequenceid=344, compaction requested=false 2023-04-13 02:56:09,049 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:09,049 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,049 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,056 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,061 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:09,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 02:56:09,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,062 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382b4b02[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:09,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:09,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:09,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:09,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:09,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:09,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,064 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,065 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,067 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,067 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,070 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,070 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,073 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,075 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354629071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,077 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354629072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354629071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354629077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,082 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,083 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:09,083 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,083 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,145 WARN [DataStreamer for file /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413fdd17dfad04f4c8486ef0a47b7166900] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-13 02:56:09,154 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413fdd17dfad04f4c8486ef0a47b7166900 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fdd17dfad04f4c8486ef0a47b7166900 2023-04-13 02:56:09,167 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/869a6550010f4bb7aee18f246878f0e1 2023-04-13 02:56:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354629178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354629178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354629178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,183 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/cb74f9a33ad04d078bd10949ed61fd68 2023-04-13 02:56:09,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354629181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,190 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,190 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,192 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,193 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:09,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,204 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,213 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,220 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,221 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,221 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,222 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,222 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c0dbe[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,224 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,224 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,225 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,225 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,225 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,225 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,225 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,228 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,228 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,228 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,228 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,230 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,231 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,231 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,232 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,234 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,235 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,237 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,238 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,238 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:09,238 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,238 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/750b62ffaa0f4da88e8a0cbf6c65cf5a 2023-04-13 02:56:09,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/869a6550010f4bb7aee18f246878f0e1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1 2023-04-13 02:56:09,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1, entries=200, sequenceid=384, filesize=32.3 K 2023-04-13 02:56:09,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/cb74f9a33ad04d078bd10949ed61fd68 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68 2023-04-13 02:56:09,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68, entries=150, sequenceid=384, filesize=11.9 K 2023-04-13 02:56:09,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/750b62ffaa0f4da88e8a0cbf6c65cf5a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a 2023-04-13 02:56:09,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a, entries=150, sequenceid=384, filesize=11.9 K 2023-04-13 02:56:09,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 63bd78eb0ff0948d99f46c700cb54493 in 211ms, sequenceid=384, compaction requested=true 2023-04-13 02:56:09,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:09,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:09,272 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:09,272 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:09,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:09,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:09,273 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:09,273 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:09,273 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:09,273 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:09,273 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,273 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,273 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/7d36889bc6194b63987538241649524d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=83.8 K 2023-04-13 02:56:09,274 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/33b4ca87a9e14cc5b505848505114303, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.3 K 2023-04-13 02:56:09,274 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 7d36889bc6194b63987538241649524d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:09,274 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 33b4ca87a9e14cc5b505848505114303, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:09,274 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 07de2cba946e4f30914981d377a08b02, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1681354567603 2023-04-13 02:56:09,274 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 96400567d3cf41da836555f5eea00d3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1681354567603 2023-04-13 02:56:09,275 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 869a6550010f4bb7aee18f246878f0e1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354568747 2023-04-13 02:56:09,275 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cb74f9a33ad04d078bd10949ed61fd68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354569060 2023-04-13 02:56:09,282 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#562 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:09,317 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#563 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:09,338 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,354 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/3d9f91fb34a84257a3d672451c6b3100 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3d9f91fb34a84257a3d672451c6b3100 2023-04-13 02:56:09,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:09,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:09,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,358 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,358 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,359 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,360 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,360 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,361 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,362 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 3d9f91fb34a84257a3d672451c6b3100(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:09,362 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:09,362 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=13, startTime=1681354569272; duration=0sec 2023-04-13 02:56:09,362 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:09,362 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:09,363 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:09,364 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:09,364 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,364 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c37396194694491a9223c26015f4adbc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.3 K 2023-04-13 02:56:09,364 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c37396194694491a9223c26015f4adbc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681354566474 2023-04-13 02:56:09,365 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 5cd089e85d684c118ebcaf312810e823, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1681354567603 2023-04-13 02:56:09,365 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 750b62ffaa0f4da88e8a0cbf6c65cf5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354569060 2023-04-13 02:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,372 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,377 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#564 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:09,400 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,400 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,400 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,400 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,400 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,400 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e742501[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,413 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,414 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,414 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,414 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,416 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,416 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354629415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354569393, max timestamp=1681354569418Num uniques 859; 2023-04-13 02:56:09,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:09,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354569393, max timestamp=1681354569418Num uniques 859; 2023-04-13 02:56:09,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:09,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681354569393, max timestamp=1681354569418Num uniques 859; 2023-04-13 02:56:09,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 02:56:09,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,426 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,426 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,426 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,428 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,429 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,429 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,431 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,435 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,435 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,435 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:09,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354629440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354629441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354629442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354629443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,471 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/f6eb35735d294c29bdd717a17f95a818 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f6eb35735d294c29bdd717a17f95a818 2023-04-13 02:56:09,473 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2fb4f748631c4d59b4b7bd655e215a07 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2fb4f748631c4d59b4b7bd655e215a07 2023-04-13 02:56:09,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into f6eb35735d294c29bdd717a17f95a818(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:09,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:09,479 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=13, startTime=1681354569272; duration=0sec 2023-04-13 02:56:09,479 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:09,482 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 2fb4f748631c4d59b4b7bd655e215a07(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:09,483 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:09,483 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=13, startTime=1681354569272; duration=0sec 2023-04-13 02:56:09,483 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:09,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354629523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,536 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,541 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,542 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:09,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,544 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354629546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354629547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354629548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354629549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,552 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,552 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,552 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,553 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,553 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,553 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,554 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,555 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,556 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321d46a2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,558 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,559 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,561 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,561 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,561 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,561 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,564 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,564 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,566 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,566 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,566 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,568 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,570 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,573 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,573 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,673 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,677 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,677 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,678 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,678 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,680 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,680 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,681 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,681 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,681 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,681 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,683 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,683 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,684 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508c78b8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,685 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,687 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,687 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,687 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,688 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,689 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,689 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,692 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,692 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,694 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,695 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,696 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,699 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,699 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,699 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,700 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,699 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354629727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354629750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354629751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354629752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354629752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,800 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,803 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,806 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,807 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,807 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,807 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,807 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,807 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,808 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,809 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,810 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e9ceea[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,811 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,819 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,820 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,820 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,821 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,821 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,821 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,821 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,824 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,824 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,824 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,824 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,824 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,825 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,826 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,830 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,830 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,830 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,830 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:09,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,846 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ed15f0ad916b448d946a86586d5d0a1b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed15f0ad916b448d946a86586d5d0a1b 2023-04-13 02:56:09,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=394, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/f8203d7eb17c411e9b0b3e09bae41967 2023-04-13 02:56:09,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/1f65759a8c344692b16f032706782049 2023-04-13 02:56:09,931 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:09,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:09,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:09,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:09,935 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:09,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:09,935 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:09,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,936 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:09,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:09,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:09,940 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:09,940 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,941 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:09,941 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:09,941 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:09,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,943 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:09,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2beab15b[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:09,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:09,947 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,949 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:09,949 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:09,950 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,950 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:09,955 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:09,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:09,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:09,955 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:09,955 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:09,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:09,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,959 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,962 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:09,966 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:09,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:09,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:09,966 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:09,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:09,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:09,967 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:09,967 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:09,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:10,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354630030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354630054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354630055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354630057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354630058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,067 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:10,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:10,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:10,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:10,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:10,070 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:10,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:10,070 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:10,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,071 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,072 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,072 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:10,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:10,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:10,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:10,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,073 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:10,074 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,074 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:10,074 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:10,074 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:10,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:10,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:10,078 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739acad2[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:10,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:10,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,081 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:10,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:10,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,082 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:10,082 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:10,085 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:10,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:10,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:10,085 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:10,085 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:10,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,086 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,087 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,087 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,088 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,090 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,090 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:10,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:10,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:10,091 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:10,091 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,191 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:10,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:10,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:10,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:10,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:10,194 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,197 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,197 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,198 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:10,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:10,199 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,199 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:10,199 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:10,200 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:10,201 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,201 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,202 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e035a1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:10,204 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:10,204 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,205 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:10,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,206 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:10,210 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,210 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:10,213 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:10,213 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:10,213 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:10,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,214 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,215 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,216 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,218 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:10,219 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,219 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:10,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,219 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,219 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:10,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/f830e7ebde2242ef887aba8632f94b72 2023-04-13 02:56:10,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/f8203d7eb17c411e9b0b3e09bae41967 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967 2023-04-13 02:56:10,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967, entries=200, sequenceid=394, filesize=32.3 K 2023-04-13 02:56:10,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/1f65759a8c344692b16f032706782049 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049 2023-04-13 02:56:10,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049, entries=150, sequenceid=394, filesize=11.9 K 2023-04-13 02:56:10,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/f830e7ebde2242ef887aba8632f94b72 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72 2023-04-13 02:56:10,319 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:10,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72, entries=150, sequenceid=394, filesize=11.9 K 2023-04-13 02:56:10,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:10,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.55 KB/134705 for 63bd78eb0ff0948d99f46c700cb54493 in 968ms, sequenceid=394, compaction requested=false 2023-04-13 02:56:10,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:10,323 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,323 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:10,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:10,326 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:10,326 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:10,327 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:10,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:10,345 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041396a56a68b38346e28910900685d6be33 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041396a56a68b38346e28910900685d6be33 2023-04-13 02:56:10,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:10,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:10,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354630557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354630559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354630562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354630562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354630566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354630662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,751 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/33592e96fb274caf885ddf71373c62c9 2023-04-13 02:56:10,763 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/4152b009fac54f259b6a0394ac44930d 2023-04-13 02:56:10,771 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/f0621aabfc7b4ccc860e1081145b8b56 2023-04-13 02:56:10,775 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/33592e96fb274caf885ddf71373c62c9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9 2023-04-13 02:56:10,779 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9, entries=150, sequenceid=433, filesize=25.4 K 2023-04-13 02:56:10,779 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/4152b009fac54f259b6a0394ac44930d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d 2023-04-13 02:56:10,782 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d, entries=150, sequenceid=433, filesize=11.9 K 2023-04-13 02:56:10,783 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/f0621aabfc7b4ccc860e1081145b8b56 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56 2023-04-13 02:56:10,786 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56, entries=150, sequenceid=433, filesize=11.9 K 2023-04-13 02:56:10,786 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for 63bd78eb0ff0948d99f46c700cb54493 in 459ms, sequenceid=433, compaction requested=true 2023-04-13 02:56:10,787 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:10,787 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:10,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:10,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:10,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:10,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,789 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:10,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a443dc9[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:10,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:10,791 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,792 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:10,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:10,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:10,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,793 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,793 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:10,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:10,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:10,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:10,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:10,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:10,795 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:10,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,797 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,798 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:10,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,803 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:10,803 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:10,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,803 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:10,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:10,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:10,903 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:10,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:10,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:10,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:10,907 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:10,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:10,907 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:10,908 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,908 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,909 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:10,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:10,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:10,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:10,917 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:10,918 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:10,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:10,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:10,923 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:10,923 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:10,925 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,925 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:10,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a3bb3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:10,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:10,928 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,929 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:10,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,929 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:10,931 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:10,934 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:10,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:10,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:10,934 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:10,934 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:10,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354570922, max timestamp=1681354570934Num uniques 859; 2023-04-13 02:56:10,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:10,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354570922, max timestamp=1681354570934Num uniques 859; 2023-04-13 02:56:10,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354570922, max timestamp=1681354570934Num uniques 859; 2023-04-13 02:56:10,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:10,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:10,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:10,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,938 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:10,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,942 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:10,942 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:10,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:10,943 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:10,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:10,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:10,943 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:10,943 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:10,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:10,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:10,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:10,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354630986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,043 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,047 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,047 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,048 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,050 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,050 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,051 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,052 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,052 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,052 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,052 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,052 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,054 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,054 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2133adb0[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,058 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,059 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,059 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,060 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,063 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,063 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,064 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,065 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,067 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:11,076 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,076 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,076 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,077 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,077 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354631089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,177 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,180 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,180 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,181 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,181 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,185 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,186 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,187 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,187 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,188 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,188 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,189 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603fc90a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,191 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,192 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,192 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,192 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,194 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,194 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,197 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,197 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,197 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,197 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,197 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,199 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,199 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,199 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,200 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,201 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,203 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,203 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,204 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,204 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,204 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,283 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137a3155ddda7a430f865f19b6fc1d55b1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137a3155ddda7a430f865f19b6fc1d55b1 2023-04-13 02:56:11,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354631293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,304 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,307 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,307 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,308 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,309 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,309 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,311 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,311 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,313 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,313 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,314 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,314 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bce0ab1[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,316 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,317 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,317 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,318 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,319 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,322 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,322 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,322 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,324 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,324 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,325 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,328 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:11,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,329 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,329 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,329 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,429 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,432 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,433 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,434 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,434 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,435 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,436 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,436 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,436 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,436 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,437 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,437 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,438 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,438 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,439 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9c39d8[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,440 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,440 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,441 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,441 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,442 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,442 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,450 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,452 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,453 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,455 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,455 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,455 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,456 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,457 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,459 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,460 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,459 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,460 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,460 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,561 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,564 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,564 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,565 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354631562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,566 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,566 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,566 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,566 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,568 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,568 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,569 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354631568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,570 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,570 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354631568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52fe4796[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,572 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,572 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,573 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,573 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354631570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,576 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,576 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,579 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,579 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,579 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,580 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,580 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,580 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,582 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,584 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,584 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,584 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,584 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:11,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,584 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,584 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,585 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,585 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,585 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:11,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354631597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,685 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,689 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,689 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,690 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=444, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ca0e74718bfc459cb788941ef71a2b01 2023-04-13 02:56:11,691 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,692 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,693 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,693 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,694 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,694 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,694 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:11,694 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,695 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,697 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,697 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,698 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,698 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c97a35a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,699 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,699 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,701 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,701 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/8325c9aeda7849d8bc7b6aa8c7763a08 2023-04-13 02:56:11,703 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,703 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,703 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,703 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,708 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,708 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,709 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,711 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/7e5129d5ee9348b29dd25c745a805b82 2023-04-13 02:56:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,713 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,714 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/ca0e74718bfc459cb788941ef71a2b01 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,719 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,720 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,720 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,720 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01, entries=200, sequenceid=444, filesize=32.3 K 2023-04-13 02:56:11,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/8325c9aeda7849d8bc7b6aa8c7763a08 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08 2023-04-13 02:56:11,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08, entries=150, sequenceid=444, filesize=11.9 K 2023-04-13 02:56:11,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/7e5129d5ee9348b29dd25c745a805b82 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82 2023-04-13 02:56:11,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82, entries=150, sequenceid=444, filesize=11.9 K 2023-04-13 02:56:11,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 63bd78eb0ff0948d99f46c700cb54493 in 862ms, sequenceid=444, compaction requested=true 2023-04-13 02:56:11,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:11,731 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:11,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:11,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:11,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:11,731 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:11,735 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 119014 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:11,735 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:11,735 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:11,735 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:11,735 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,735 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,735 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2fb4f748631c4d59b4b7bd655e215a07, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=116.2 K 2023-04-13 02:56:11,735 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3d9f91fb34a84257a3d672451c6b3100, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.3 K 2023-04-13 02:56:11,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2fb4f748631c4d59b4b7bd655e215a07, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354569060 2023-04-13 02:56:11,736 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 3d9f91fb34a84257a3d672451c6b3100, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354569060 2023-04-13 02:56:11,736 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f8203d7eb17c411e9b0b3e09bae41967, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681354569070 2023-04-13 02:56:11,740 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 33592e96fb274caf885ddf71373c62c9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681354569433 2023-04-13 02:56:11,740 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 1f65759a8c344692b16f032706782049, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681354569071 2023-04-13 02:56:11,740 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting ca0e74718bfc459cb788941ef71a2b01, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:11,747 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 4152b009fac54f259b6a0394ac44930d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681354569433 2023-04-13 02:56:11,748 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 8325c9aeda7849d8bc7b6aa8c7763a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:11,762 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#575 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:11,777 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#574 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:11,792 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/60b87f9fa77045ed8b9ee8924ec4ab36 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/60b87f9fa77045ed8b9ee8924ec4ab36 2023-04-13 02:56:11,796 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 60b87f9fa77045ed8b9ee8924ec4ab36(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:11,796 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:11,796 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=12, startTime=1681354571731; duration=0sec 2023-04-13 02:56:11,796 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:11,796 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:11,798 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:11,798 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:11,798 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,798 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f6eb35735d294c29bdd717a17f95a818, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.3 K 2023-04-13 02:56:11,799 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f6eb35735d294c29bdd717a17f95a818, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681354569060 2023-04-13 02:56:11,799 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f830e7ebde2242ef887aba8632f94b72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681354569071 2023-04-13 02:56:11,801 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f0621aabfc7b4ccc860e1081145b8b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681354569433 2023-04-13 02:56:11,801 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 7e5129d5ee9348b29dd25c745a805b82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:11,810 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#576 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:11,820 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:11,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:11,821 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/585f612cee24408c8395f10e3f034baa as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/585f612cee24408c8395f10e3f034baa 2023-04-13 02:56:11,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:11,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:11,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:11,825 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:11,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:11,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:11,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,825 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,826 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,827 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 585f612cee24408c8395f10e3f034baa(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:11,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,827 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:11,827 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=12, startTime=1681354571731; duration=0sec 2023-04-13 02:56:11,827 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:11,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,827 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:11,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:11,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:11,828 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:11,828 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:11,828 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:11,828 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:11,828 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,828 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:11,828 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:11,829 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:11,829 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:11,848 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d4b47bb423cf4666b371b27bf4dcd961 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4b47bb423cf4666b371b27bf4dcd961 2023-04-13 02:56:11,857 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=482, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2afccbca15784144b27c0a43197e5410 2023-04-13 02:56:11,867 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/489e5aaa64b7496a83f7627e46cfb508 2023-04-13 02:56:11,877 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/fb5c0b2c29c84a068a1c3152b616a3a0 2023-04-13 02:56:11,881 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2afccbca15784144b27c0a43197e5410 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410 2023-04-13 02:56:11,884 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410, entries=150, sequenceid=482, filesize=25.4 K 2023-04-13 02:56:11,885 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/489e5aaa64b7496a83f7627e46cfb508 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508 2023-04-13 02:56:11,888 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508, entries=150, sequenceid=482, filesize=11.9 K 2023-04-13 02:56:11,889 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/fb5c0b2c29c84a068a1c3152b616a3a0 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0 2023-04-13 02:56:11,892 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0, entries=150, sequenceid=482, filesize=11.9 K 2023-04-13 02:56:11,893 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 63bd78eb0ff0948d99f46c700cb54493 in 64ms, sequenceid=482, compaction requested=false 2023-04-13 02:56:11,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:11,893 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:11,893 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:11,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,909 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:11,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:11,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7b3fb[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:11,910 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:11,911 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:11,911 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:11,911 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,911 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:11,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:11,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:11,916 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:11,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:11,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:11,916 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:11,916 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:11,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:11,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,919 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,920 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:11,922 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:11,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:11,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:11,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:11,922 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:11,923 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:11,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:11,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:11,923 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:11,923 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:11,923 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,023 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,026 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,037 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:12,038 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,038 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,039 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,040 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,040 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,041 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,041 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@81a0822[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,044 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,044 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,045 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,046 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,046 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,049 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,049 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,049 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,052 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,052 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,054 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,057 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,057 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:12,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,057 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,157 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,160 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,162 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,164 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,164 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,164 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:12,165 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:12,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:12,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,181 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041324fe23e16c9e45249333f73882eba9a9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fe23e16c9e45249333f73882eba9a9 2023-04-13 02:56:12,193 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=492, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/421182b9d6e64405b36bf94f94183343 2023-04-13 02:56:12,209 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/0872cafcc9444caa9be09c0097ae5a90 2023-04-13 02:56:12,217 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/9450ad520d8b46c4a66f458c9f6bb48f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9450ad520d8b46c4a66f458c9f6bb48f 2023-04-13 02:56:12,219 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/820f41ee1e5c4e90b509179634dc6b0d 2023-04-13 02:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572195, max timestamp=1681354572220Num uniques 859; 2023-04-13 02:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572195, max timestamp=1681354572220Num uniques 859; 2023-04-13 02:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,223 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 9450ad520d8b46c4a66f458c9f6bb48f(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:12,223 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:12,223 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=12, startTime=1681354571731; duration=0sec 2023-04-13 02:56:12,223 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:12,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572195, max timestamp=1681354572220Num uniques 859; 2023-04-13 02:56:12,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,225 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/421182b9d6e64405b36bf94f94183343 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343 2023-04-13 02:56:12,229 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343, entries=150, sequenceid=492, filesize=25.4 K 2023-04-13 02:56:12,229 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/0872cafcc9444caa9be09c0097ae5a90 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90 2023-04-13 02:56:12,232 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90, entries=150, sequenceid=492, filesize=11.9 K 2023-04-13 02:56:12,233 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/820f41ee1e5c4e90b509179634dc6b0d as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d 2023-04-13 02:56:12,237 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d, entries=150, sequenceid=492, filesize=11.9 K 2023-04-13 02:56:12,238 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=53.67 KB/54960 for 63bd78eb0ff0948d99f46c700cb54493 in 74ms, sequenceid=492, compaction requested=true 2023-04-13 02:56:12,238 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:12,238 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:12,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-04-13 02:56:12,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:12,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:12,238 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:12,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:12,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:12,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:12,240 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,240 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28783c40[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,244 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,251 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,251 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,251 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,251 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,252 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,253 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,256 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,256 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,256 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,258 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,258 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,258 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,259 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,261 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,263 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,263 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,263 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,263 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,263 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,264 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,264 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,264 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572294, max timestamp=1681354572308Num uniques 859; 2023-04-13 02:56:12,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572294, max timestamp=1681354572308Num uniques 859; 2023-04-13 02:56:12,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354572294, max timestamp=1681354572308Num uniques 859; 2023-04-13 02:56:12,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:12,364 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,368 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,368 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:12,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354632367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,374 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,374 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,375 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,376 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,377 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,377 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,377 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,377 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,379 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5904d766[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,381 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,381 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,382 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,382 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,383 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,384 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,387 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,387 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,387 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,389 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,389 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,389 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,390 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,391 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,394 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,394 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,395 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:12,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354632471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,495 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,498 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,498 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,499 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,499 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,500 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,500 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,501 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,501 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,501 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,501 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,503 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,503 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e6efd6[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,505 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,505 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,509 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,509 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,509 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,510 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,513 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,513 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,513 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,513 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,513 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,514 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,514 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,514 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,515 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,516 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,519 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,519 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,519 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,519 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,619 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,623 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,623 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,624 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,628 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,628 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,631 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b65b090[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,633 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,634 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,634 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,634 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,634 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,634 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,636 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,638 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,638 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,638 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,641 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,641 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,641 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,643 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,645 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,645 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,645 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,646 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:12,677 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041392f15237f48d40e7976459d56e1d7ad5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041392f15237f48d40e7976459d56e1d7ad5 2023-04-13 02:56:12,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:12,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354632674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,746 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,749 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,749 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,750 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,750 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,751 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,752 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,752 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,753 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,753 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,754 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,754 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,755 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52961180[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,758 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,758 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,764 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,764 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,764 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,766 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,766 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,769 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,769 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,769 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,771 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,771 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,771 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,772 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,773 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,781 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,781 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,781 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,781 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,882 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:12,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:12,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:12,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:12,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:12,885 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:12,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:12,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:12,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,885 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,890 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,890 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:12,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:12,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:12,891 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:12,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:12,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:12,892 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:12,892 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:12,894 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,894 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,895 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:12,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d564353[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:12,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:12,896 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,897 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:12,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:12,898 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:12,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,898 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:12,902 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:12,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,902 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:12,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:12,905 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:12,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:12,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:12,905 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:12,905 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:12,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:12,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:12,909 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,911 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,914 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:12,914 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:12,914 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:12,914 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:12,915 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:12,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:12,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:12,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354632980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,015 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,018 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,018 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,020 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,020 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,021 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,022 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,022 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,024 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,024 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791a8170[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,026 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,027 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,027 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,028 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,030 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,030 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,030 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,033 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,033 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,034 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,036 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,037 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,037 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,037 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,083 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=519, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/f1ea37a9ac544bb1bc17634074cdf7bf 2023-04-13 02:56:13,137 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,141 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,141 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,142 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,142 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,143 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,145 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,146 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,146 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,147 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,147 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,149 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fed4ff3[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,153 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,153 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,155 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,157 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,157 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,157 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,159 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,159 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,160 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,163 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,163 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,163 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,163 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,263 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,267 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,267 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,268 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,268 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,270 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,270 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,272 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,274 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,274 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,276 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,277 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d15599f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,281 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,281 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,283 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,286 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,286 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,289 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,289 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,289 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,290 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,291 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,297 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,298 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,298 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,298 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,398 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,401 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,401 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,402 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,402 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,403 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,403 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,405 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,405 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,405 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,405 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,406 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,407 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c6cfb5[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,409 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,411 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,411 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,411 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,413 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,415 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,415 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,416 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,417 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,418 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,419 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,421 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,421 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,422 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,422 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,422 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:13,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354633488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/f0d6a9ecefc34ea7be1a5e19ef59e879 2023-04-13 02:56:13,522 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,525 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,525 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,528 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,529 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,531 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,531 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffacd9c[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,536 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,536 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,537 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,540 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,540 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,542 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,543 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,543 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,545 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,547 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,547 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,547 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,547 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:13,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354633576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:13,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354633576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,580 DEBUG [Thread-3697] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:13,581 DEBUG [Thread-3693] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:13,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:13,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354633578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:13,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354633578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,585 DEBUG [Thread-3699] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:13,586 DEBUG [Thread-3701] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:13,648 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,652 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,652 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,654 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,654 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,657 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,657 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,659 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,660 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,661 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aed9f47[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,662 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,664 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,664 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,666 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,666 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,669 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,669 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,669 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,671 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,671 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,672 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,679 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,679 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,680 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,680 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,680 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,780 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,784 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,784 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,785 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,788 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,790 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,790 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,791 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d93e024[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,792 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,794 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,794 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,795 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,794 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,795 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,796 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,796 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,799 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,799 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,799 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,800 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,801 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,801 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,803 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,805 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,805 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,806 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,806 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,806 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/32f05e000a474ced90f361dda97af275 2023-04-13 02:56:13,906 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:13,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/f1ea37a9ac544bb1bc17634074cdf7bf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf 2023-04-13 02:56:13,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:13,910 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:13,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:13,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:13,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,911 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf, entries=200, sequenceid=519, filesize=32.3 K 2023-04-13 02:56:13,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/f0d6a9ecefc34ea7be1a5e19ef59e879 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879 2023-04-13 02:56:13,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879, entries=150, sequenceid=519, filesize=11.9 K 2023-04-13 02:56:13,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/32f05e000a474ced90f361dda97af275 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275 2023-04-13 02:56:13,918 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,918 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:13,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:13,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:13,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:13,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:13,919 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:13,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:13,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:13,920 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:13,920 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:13,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275, entries=150, sequenceid=519, filesize=11.9 K 2023-04-13 02:56:13,922 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 63bd78eb0ff0948d99f46c700cb54493 in 1684ms, sequenceid=519, compaction requested=true 2023-04-13 02:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,922 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:13,937 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:13,922 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:13,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112110 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:13,939 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,939 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:13,939 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9450ad520d8b46c4a66f458c9f6bb48f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=109.5 K 2023-04-13 02:56:13,939 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/60b87f9fa77045ed8b9ee8924ec4ab36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.5 K 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9450ad520d8b46c4a66f458c9f6bb48f, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:13,939 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 60b87f9fa77045ed8b9ee8924ec4ab36, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,940 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2afccbca15784144b27c0a43197e5410, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681354570960 2023-04-13 02:56:13,940 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 489e5aaa64b7496a83f7627e46cfb508, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681354570960 2023-04-13 02:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:13,940 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 421182b9d6e64405b36bf94f94183343, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681354572155 2023-04-13 02:56:13,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398c2f44[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:13,940 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,940 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 0872cafcc9444caa9be09c0097ae5a90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681354572155 2023-04-13 02:56:13,941 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f1ea37a9ac544bb1bc17634074cdf7bf, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:13,941 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting f0d6a9ecefc34ea7be1a5e19ef59e879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:13,942 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:13,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:13,942 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,944 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:13,944 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:13,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:13,948 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:13,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:13,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:13,949 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:13,949 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:13,949 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#586 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:13,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,951 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:13,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:13,952 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:13,954 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:13,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:13,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:13,957 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#587 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:13,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:13,956 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:13,957 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:13,958 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:13,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:13,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:13,956 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:13,958 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:13,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:14,058 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:14,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:14,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:14,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:14,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:14,062 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:14,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:14,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:14,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:14,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,062 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:14,063 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:14,063 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:14,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,064 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:14,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:14,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:14,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:14,066 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:14,066 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:14,066 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:14,067 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:14,365 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/2d6c6f24c36c4720be3ca13e95ee772a as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2d6c6f24c36c4720be3ca13e95ee772a 2023-04-13 02:56:14,369 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 2d6c6f24c36c4720be3ca13e95ee772a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:14,369 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:14,369 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=12, startTime=1681354573922; duration=0sec 2023-04-13 02:56:14,369 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:14,369 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:14,371 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:14,371 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:14,371 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:14,371 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/585f612cee24408c8395f10e3f034baa, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.5 K 2023-04-13 02:56:14,372 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 585f612cee24408c8395f10e3f034baa, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681354570550 2023-04-13 02:56:14,372 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting fb5c0b2c29c84a068a1c3152b616a3a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681354570960 2023-04-13 02:56:14,372 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 820f41ee1e5c4e90b509179634dc6b0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681354572155 2023-04-13 02:56:14,373 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 32f05e000a474ced90f361dda97af275, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:14,378 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#589 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:14,386 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/932df346dd5a4369aa73e0dc33107ff6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/932df346dd5a4369aa73e0dc33107ff6 2023-04-13 02:56:14,387 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/65858cfbb8b14e01810f49f1901a4525 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/65858cfbb8b14e01810f49f1901a4525 2023-04-13 02:56:14,390 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into 932df346dd5a4369aa73e0dc33107ff6(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:14,390 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:14,390 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=12, startTime=1681354573922; duration=0sec 2023-04-13 02:56:14,390 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:14,391 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 65858cfbb8b14e01810f49f1901a4525(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:14,391 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:14,391 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=12, startTime=1681354573937; duration=0sec 2023-04-13 02:56:14,391 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:14,481 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ed8fcbdf780844cf96450d59674db47e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed8fcbdf780844cf96450d59674db47e 2023-04-13 02:56:14,486 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=555, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/cc8a752adcac433886c363ceb48d8ba9 2023-04-13 02:56:14,494 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/cc3cfc0e9ac24f819f5b93455574ecda 2023-04-13 02:56:14,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:14,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:14,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:14,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354634525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:14,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354634627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:14,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354634831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,907 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/ae15b071262e4ff4a3749f8bf8a63bbf 2023-04-13 02:56:14,911 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/cc8a752adcac433886c363ceb48d8ba9 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9 2023-04-13 02:56:14,916 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9, entries=150, sequenceid=555, filesize=25.4 K 2023-04-13 02:56:14,917 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/cc3cfc0e9ac24f819f5b93455574ecda as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda 2023-04-13 02:56:14,920 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda, entries=150, sequenceid=555, filesize=11.9 K 2023-04-13 02:56:14,921 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/ae15b071262e4ff4a3749f8bf8a63bbf as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf 2023-04-13 02:56:14,925 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf, entries=150, sequenceid=555, filesize=11.9 K 2023-04-13 02:56:14,926 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 63bd78eb0ff0948d99f46c700cb54493 in 860ms, sequenceid=555, compaction requested=false 2023-04-13 02:56:14,926 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:14,926 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:14,926 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:14,928 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:14,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,929 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:14,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507393c4[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:14,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:14,930 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,932 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:14,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,933 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:14,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:14,934 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:14,934 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:14,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:14,939 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:14,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:14,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:14,939 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:14,939 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:14,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,941 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:14,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,942 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,943 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:14,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:14,946 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:14,946 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:14,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:14,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:14,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:14,947 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:14,947 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:14,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:15,047 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:15,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:15,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:15,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:15,051 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,051 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,056 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,056 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:15,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:15,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:15,057 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:15,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:15,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:15,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,058 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:15,058 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:15,059 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:15,059 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:15,078 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413efd7ba6990de419fa76d956ab23b237d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413efd7ba6990de419fa76d956ab23b237d 2023-04-13 02:56:15,084 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=569, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/4998b1555cce48369896924064b7a399 2023-04-13 02:56:15,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:15,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354575156, max timestamp=1681354575204Num uniques 859; 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354575156, max timestamp=1681354575204Num uniques 859; 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681354575156, max timestamp=1681354575204Num uniques 859; 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:15,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 02:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354635246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354635350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,493 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/49da5ecba1644be4984d865d5a499998 2023-04-13 02:56:15,502 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/34719fb205144208b2f48f95bdb2ca46 2023-04-13 02:56:15,506 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/4998b1555cce48369896924064b7a399 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399 2023-04-13 02:56:15,509 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399, entries=150, sequenceid=569, filesize=25.4 K 2023-04-13 02:56:15,510 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/49da5ecba1644be4984d865d5a499998 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998 2023-04-13 02:56:15,513 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998, entries=150, sequenceid=569, filesize=11.9 K 2023-04-13 02:56:15,514 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/34719fb205144208b2f48f95bdb2ca46 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46 2023-04-13 02:56:15,518 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46, entries=150, sequenceid=569, filesize=11.9 K 2023-04-13 02:56:15,519 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 63bd78eb0ff0948d99f46c700cb54493 in 460ms, sequenceid=569, compaction requested=true 2023-04-13 02:56:15,519 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:15,519 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:15,520 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,520 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,521 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632292a[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:15,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:15,522 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,523 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:15,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,524 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:15,526 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,526 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:15,528 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:15,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:15,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:15,528 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:15,528 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:15,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,530 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,530 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,532 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,535 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:15,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,534 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,535 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,535 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:15,535 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:15,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:15,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:15,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 02:56:15,567 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304135c540ecae014479dac31d81929d4efe9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c540ecae014479dac31d81929d4efe9 2023-04-13 02:56:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354635582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,635 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:15,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:15,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:15,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:15,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:15,639 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:15,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:15,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:15,639 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,640 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,641 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,641 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:15,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:15,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:15,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:15,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,643 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:15,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:15,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:15,644 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:15,647 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,647 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,648 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,648 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a27b08[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:15,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:15,649 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,650 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:15,650 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:15,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,651 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:15,652 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,652 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:15,655 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:15,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:15,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:15,655 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:15,655 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:15,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,657 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,658 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,659 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,663 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,663 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,664 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:15,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:15,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:15,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,664 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:15,664 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:15,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354635685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,764 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:15,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:15,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:15,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:15,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:15,768 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:15,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:15,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:15,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,768 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,770 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,770 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:15,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:15,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:15,771 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:15,772 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,772 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:15,772 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:15,772 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:15,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:15,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:15,772 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:15,774 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,774 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,774 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,774 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:15,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0c82d7[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:15,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:15,775 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,777 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:15,777 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:15,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,777 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:15,779 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,779 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:15,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:15,782 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:15,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:15,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:15,782 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:15,782 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:15,784 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,784 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,784 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,785 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,786 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,788 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:15,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:15,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:15,789 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:15,789 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:15,788 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,789 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,889 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:15,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:15,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:15,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354635888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:15,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:15,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:15,892 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:15,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:15,892 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:15,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,893 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,895 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,895 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:15,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:15,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:15,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:15,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:15,897 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,900 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102dd721[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:15,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:15,902 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,905 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:15,906 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:15,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,906 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:15,907 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,908 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:15,910 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:15,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:15,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:15,910 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:15,911 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:15,912 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:15,913 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,914 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:15,917 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:15,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:15,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:15,917 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:15,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:15,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:15,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:15,918 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:15,918 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:15,972 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=605, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/4e3b7824b36a40acae4b9b04c7891f02 2023-04-13 02:56:16,018 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:16,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:16,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:16,022 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:16,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:16,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:16,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,022 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,025 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,025 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:16,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:16,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:16,027 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,027 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:16,027 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:16,027 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:16,029 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:16,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1c777[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:16,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:16,031 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,032 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:16,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:16,032 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:16,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,032 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,033 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:16,036 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:16,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:16,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:16,036 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:16,036 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,038 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,039 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,040 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,043 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:16,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,043 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,043 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,043 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:16,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:16,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,144 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:16,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:16,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:16,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:16,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:16,147 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:16,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:16,147 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:16,148 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,148 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,149 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,149 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,150 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,150 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:16,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:16,151 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:16,151 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,152 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:16,152 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:16,152 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:16,154 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,154 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,155 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac86bc[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:16,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:16,156 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,161 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:16,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,161 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:16,162 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:16,166 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:16,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:16,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:16,166 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:16,166 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,167 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,168 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,169 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,171 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:16,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,171 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,172 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:16,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,172 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,172 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354636197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,272 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:16,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:16,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:16,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:16,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:16,275 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:16,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:16,275 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:16,276 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,276 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,278 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,278 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,279 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:16,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:16,280 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:16,280 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:16,280 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:16,280 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:16,281 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:16,282 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288ea444[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:16,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:16,284 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,286 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:16,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,286 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:16,291 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:16,294 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:16,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:16,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:16,294 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:16,294 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:16,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,296 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:16,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,297 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,298 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:16,300 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:16,301 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,301 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:16,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:16,301 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:16,301 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:16,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:16,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/18358ed45719454a8e9c8a26ed49efa2 2023-04-13 02:56:16,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/890eb5786e7645ccb4722861901e0071 2023-04-13 02:56:16,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/4e3b7824b36a40acae4b9b04c7891f02 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02 2023-04-13 02:56:16,399 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02, entries=150, sequenceid=605, filesize=25.4 K 2023-04-13 02:56:16,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/18358ed45719454a8e9c8a26ed49efa2 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2 2023-04-13 02:56:16,401 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:16,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:16,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2, entries=150, sequenceid=605, filesize=11.9 K 2023-04-13 02:56:16,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/890eb5786e7645ccb4722861901e0071 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071 2023-04-13 02:56:16,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:16,405 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,405 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,407 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:16,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:16,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071, entries=150, sequenceid=605, filesize=11.9 K 2023-04-13 02:56:16,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,407 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 63bd78eb0ff0948d99f46c700cb54493 in 853ms, sequenceid=605, compaction requested=true 2023-04-13 02:56:16,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:16,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:16,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:16,408 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:16,408 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:16,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:16,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:16,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:16,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:16,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:16,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:16,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:16,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:16,409 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:16,409 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,409 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:16,409 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105206 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:16,409 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:16,410 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:16,410 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:16,410 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,410 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,410 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/932df346dd5a4369aa73e0dc33107ff6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=102.7 K 2023-04-13 02:56:16,410 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2d6c6f24c36c4720be3ca13e95ee772a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.6 K 2023-04-13 02:56:16,410 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 932df346dd5a4369aa73e0dc33107ff6, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:16,410 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 2d6c6f24c36c4720be3ca13e95ee772a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:16,410 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:56:16,411 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting cc8a752adcac433886c363ceb48d8ba9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681354572337 2023-04-13 02:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:16,411 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting cc3cfc0e9ac24f819f5b93455574ecda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681354572337 2023-04-13 02:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:16,411 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4998b1555cce48369896924064b7a399, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681354574515 2023-04-13 02:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:16,411 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 49da5ecba1644be4984d865d5a499998, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681354574515 2023-04-13 02:56:16,411 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 4e3b7824b36a40acae4b9b04c7891f02, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:16,412 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 18358ed45719454a8e9c8a26ed49efa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:16,412 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:16,426 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#600 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:16,469 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#599 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:16,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:16,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354576722, max timestamp=1681354576755Num uniques 859; 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354576722, max timestamp=1681354576755Num uniques 859; 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681354576722, max timestamp=1681354576755Num uniques 859; 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 02:56:16,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:16,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354636829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:16,861 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137d19d2fa658447a3ac83cedf03c6d5ba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137d19d2fa658447a3ac83cedf03c6d5ba 2023-04-13 02:56:16,867 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=615, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/dd8ff15987634d2d802a00614a6d2674 2023-04-13 02:56:16,869 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/894c05120f944b2ca91ab8c814bbc2c7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/894c05120f944b2ca91ab8c814bbc2c7 2023-04-13 02:56:16,876 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into 894c05120f944b2ca91ab8c814bbc2c7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:16,876 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:16,876 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=12, startTime=1681354576408; duration=0sec 2023-04-13 02:56:16,876 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:16,877 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 02:56:16,878 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 02:56:16,878 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:16,878 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:16,878 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/65858cfbb8b14e01810f49f1901a4525, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=48.6 K 2023-04-13 02:56:16,879 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 65858cfbb8b14e01810f49f1901a4525, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1681354572223 2023-04-13 02:56:16,879 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting ae15b071262e4ff4a3749f8bf8a63bbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681354572337 2023-04-13 02:56:16,879 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 34719fb205144208b2f48f95bdb2ca46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1681354574515 2023-04-13 02:56:16,880 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 890eb5786e7645ccb4722861901e0071, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:16,881 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/e6c0d57773544bc5844a4bdfd0007378 2023-04-13 02:56:16,887 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#602 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:16,890 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/f65a740889aa4c929d1ad6370f1373fb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f65a740889aa4c929d1ad6370f1373fb 2023-04-13 02:56:16,893 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into f65a740889aa4c929d1ad6370f1373fb(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:16,893 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:16,894 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=12, startTime=1681354576408; duration=0sec 2023-04-13 02:56:16,894 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:16,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354636933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354637137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,310 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/9d8ef84501fe45af885e7f5d1d3609bd 2023-04-13 02:56:17,310 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/82b45424697847bdbb4f23f31a7da072 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/82b45424697847bdbb4f23f31a7da072 2023-04-13 02:56:17,315 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/dd8ff15987634d2d802a00614a6d2674 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674 2023-04-13 02:56:17,316 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 82b45424697847bdbb4f23f31a7da072(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:17,316 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:17,316 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=12, startTime=1681354576408; duration=0sec 2023-04-13 02:56:17,316 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:17,318 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674, entries=150, sequenceid=615, filesize=25.4 K 2023-04-13 02:56:17,319 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/e6c0d57773544bc5844a4bdfd0007378 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378 2023-04-13 02:56:17,322 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378, entries=150, sequenceid=615, filesize=11.9 K 2023-04-13 02:56:17,323 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/9d8ef84501fe45af885e7f5d1d3609bd as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd 2023-04-13 02:56:17,326 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd, entries=150, sequenceid=615, filesize=11.9 K 2023-04-13 02:56:17,327 INFO [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 63bd78eb0ff0948d99f46c700cb54493 in 918ms, sequenceid=615, compaction requested=false 2023-04-13 02:56:17,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:17,327 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:17,327 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,328 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,329 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e4eaab[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:17,330 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:17,330 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,332 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:17,332 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:17,332 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,332 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:17,333 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:17,333 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,333 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:17,333 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:17,337 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:17,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:17,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:17,337 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:17,337 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:17,340 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,340 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,340 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,341 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,342 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,344 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,345 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,345 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,345 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 02:56:17,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:17,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:17,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,345 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:17,345 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:17,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:17,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:17,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 02:56:17,445 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:17,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:17,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:17,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:17,450 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:17,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:17,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:17,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:17,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,450 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,452 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:17,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,452 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,453 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:17,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:17,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:17,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:17,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:17,454 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:17,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:17,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:17,455 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:17,457 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,457 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,459 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e2d361a09eba462a8cc961a0dbbbcb19 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2d361a09eba462a8cc961a0dbbbcb19 2023-04-13 02:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f17757f[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:17,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:17,460 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,462 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:17,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:17,462 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,464 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:17,464 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,464 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=656, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2bdc3a56393243d09e25a65248db97a7 2023-04-13 02:56:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:17,469 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:17,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:17,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:17,469 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:17,469 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:17,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,471 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,472 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,473 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34919] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354637471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,475 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,476 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,476 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:17,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:17,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:17,476 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:17,476 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:17,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/c23f35b2f9674115b598d66a08ab00d3 2023-04-13 02:56:17,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,576 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:17,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:17,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 02:56:17,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354637576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 02:56:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 02:56:17,580 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 02:56:17,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:17,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 02:56:17,580 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 02:56:17,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,581 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,583 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:17,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:17,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,583 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 02:56:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 02:56:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 02:56:17,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 02:56:17,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 02:56:17,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. started... 2023-04-13 02:56:17,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. as already flushing 2023-04-13 02:56:17,585 DEBUG [rs(jenkins-hbase4.apache.org,34919,1681354416534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 02:56:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 02:56:17,587 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,587 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 02:56:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34919,1681354416534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 02:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfa9147[Count = 0] remaining members to acquire global barrier 2023-04-13 02:56:17,589 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 02:56:17,589 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,590 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 02:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 02:56:17,590 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34919,1681354416534' in zk 2023-04-13 02:56:17,590 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,591 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 02:56:17,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 02:56:17,592 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,592 DEBUG [member: 'jenkins-hbase4.apache.org,34919,1681354416534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 02:56:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34919,1681354416534': 2023-04-13 02:56:17,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34919,1681354416534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 02:56:17,594 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 02:56:17,594 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 02:56:17,595 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 02:56:17,595 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 02:56:17,595 INFO [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 02:56:17,597 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,597 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 02:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 02:56:17,597 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 02:56:17,598 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 02:56:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,599 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 02:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 02:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,604 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 02:56:17,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 02:56:17,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 02:56:17,605 DEBUG [(jenkins-hbase4.apache.org,37305,1681354415362)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 02:56:17,605 DEBUG [Thread-3703] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 02:56:17,605 DEBUG [Thread-3703] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 02:56:17,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37700 deadline: 1681354637602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,605 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 02:56:17,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 02:56:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 02:56:17,607 DEBUG [Thread-3697] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:17,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 02:56:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 02:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37702 deadline: 1681354637605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,611 DEBUG [Thread-3699] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:17,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37714 deadline: 1681354637614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,616 DEBUG [Thread-3701] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:17,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34919] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37678 deadline: 1681354637618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,623 DEBUG [Thread-3693] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., hostname=jenkins-hbase4.apache.org,34919,1681354416534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 02:56:17,639 DEBUG [Thread-3706] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:62948 2023-04-13 02:56:17,639 DEBUG [Thread-3708] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26d7d0b5 to 127.0.0.1:62948 2023-04-13 02:56:17,639 DEBUG [Thread-3706] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:17,639 DEBUG [Thread-3708] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:17,640 DEBUG [Thread-3710] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x601410a4 to 127.0.0.1:62948 2023-04-13 02:56:17,640 DEBUG [Thread-3710] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:17,645 DEBUG [Thread-3712] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f8e4d5e to 127.0.0.1:62948 2023-04-13 02:56:17,645 DEBUG [Thread-3712] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:17,645 DEBUG [Thread-3704] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:62948 2023-04-13 02:56:17,645 DEBUG [Thread-3704] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:17,705 DEBUG [Thread-3703] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 02:56:17,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 02:56:17,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 02:56:17,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37692 deadline: 1681354637780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:17,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/41d1e2d816354428831352993ba443d1 2023-04-13 02:56:17,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/2bdc3a56393243d09e25a65248db97a7 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7 2023-04-13 02:56:17,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7, entries=200, sequenceid=656, filesize=32.3 K 2023-04-13 02:56:17,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/c23f35b2f9674115b598d66a08ab00d3 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3 2023-04-13 02:56:17,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3, entries=150, sequenceid=656, filesize=11.9 K 2023-04-13 02:56:17,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/41d1e2d816354428831352993ba443d1 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1 2023-04-13 02:56:17,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1, entries=150, sequenceid=656, filesize=11.9 K 2023-04-13 02:56:17,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 63bd78eb0ff0948d99f46c700cb54493 in 455ms, sequenceid=656, compaction requested=true 2023-04-13 02:56:17,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:17,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:17,898 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:17,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:17,898 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:17,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:17,899 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86336 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:17,899 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:17,899 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/A is initiating minor compaction (all files) 2023-04-13 02:56:17,899 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/B is initiating minor compaction (all files) 2023-04-13 02:56:17,899 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/B in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,899 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/A in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,899 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/894c05120f944b2ca91ab8c814bbc2c7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.8 K 2023-04-13 02:56:17,899 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f65a740889aa4c929d1ad6370f1373fb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=84.3 K 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting 894c05120f944b2ca91ab8c814bbc2c7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting f65a740889aa4c929d1ad6370f1373fb, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting e6c0d57773544bc5844a4bdfd0007378, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681354575566 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting dd8ff15987634d2d802a00614a6d2674, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681354575566 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] compactions.Compactor(207): Compacting c23f35b2f9674115b598d66a08ab00d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1681354576812 2023-04-13 02:56:17,900 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 2bdc3a56393243d09e25a65248db97a7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1681354576776 2023-04-13 02:56:17,908 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#B#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:17,910 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#A#compaction#607 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:17,922 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/bacdb3ccab494385b0701bf58a101330 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bacdb3ccab494385b0701bf58a101330 2023-04-13 02:56:17,926 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/A of 63bd78eb0ff0948d99f46c700cb54493 into bacdb3ccab494385b0701bf58a101330(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:17,926 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:17,926 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/A, priority=13, startTime=1681354577898; duration=0sec 2023-04-13 02:56:17,926 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 02:56:17,926 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 02:56:17,926 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 02:56:17,927 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1912): 63bd78eb0ff0948d99f46c700cb54493/C is initiating minor compaction (all files) 2023-04-13 02:56:17,927 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63bd78eb0ff0948d99f46c700cb54493/C in TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:17,927 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/82b45424697847bdbb4f23f31a7da072, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1] into tmpdir=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp, totalSize=36.8 K 2023-04-13 02:56:17,927 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 82b45424697847bdbb4f23f31a7da072, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1681354575240 2023-04-13 02:56:17,927 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 9d8ef84501fe45af885e7f5d1d3609bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681354575566 2023-04-13 02:56:17,927 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] compactions.Compactor(207): Compacting 41d1e2d816354428831352993ba443d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1681354576812 2023-04-13 02:56:17,931 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63bd78eb0ff0948d99f46c700cb54493#C#compaction#609 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-13 02:56:17,938 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/7e6f459369c84276b459cae28afe05ce as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e6f459369c84276b459cae28afe05ce 2023-04-13 02:56:17,942 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/C of 63bd78eb0ff0948d99f46c700cb54493 into 7e6f459369c84276b459cae28afe05ce(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:17,942 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:17,942 INFO [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/C, priority=13, startTime=1681354577898; duration=0sec 2023-04-13 02:56:17,942 DEBUG [RS:0;jenkins-hbase4:34919-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34919] regionserver.HRegion(9158): Flush requested on 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:18,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:18,083 DEBUG [Thread-3695] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01d207bb to 127.0.0.1:62948 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:18,083 DEBUG [Thread-3695] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:18,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:18,093 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413613f927793b24ae18c8ffbbf1ca41bf3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413613f927793b24ae18c8ffbbf1ca41bf3 2023-04-13 02:56:18,315 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/a3f92a8b66684f6fb676c0211c3dddfe as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a3f92a8b66684f6fb676c0211c3dddfe 2023-04-13 02:56:18,319 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63bd78eb0ff0948d99f46c700cb54493/B of 63bd78eb0ff0948d99f46c700cb54493 into a3f92a8b66684f6fb676c0211c3dddfe(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 02:56:18,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:18,319 INFO [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493., storeName=63bd78eb0ff0948d99f46c700cb54493/B, priority=13, startTime=1681354577898; duration=0sec 2023-04-13 02:56:18,319 DEBUG [RS:0;jenkins-hbase4:34919-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 02:56:18,497 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=668, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/bb095a6df472417babaa1823ad456161 2023-04-13 02:56:18,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/2f880b99fe1e4e2b82048fe5acf0098e 2023-04-13 02:56:19,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/ab6805f01d63426b825263ea0a3e1917 2023-04-13 02:56:19,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/bb095a6df472417babaa1823ad456161 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bb095a6df472417babaa1823ad456161 2023-04-13 02:56:19,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bb095a6df472417babaa1823ad456161, entries=150, sequenceid=668, filesize=25.4 K 2023-04-13 02:56:19,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/2f880b99fe1e4e2b82048fe5acf0098e as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2f880b99fe1e4e2b82048fe5acf0098e 2023-04-13 02:56:19,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2f880b99fe1e4e2b82048fe5acf0098e, entries=150, sequenceid=668, filesize=11.9 K 2023-04-13 02:56:19,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/ab6805f01d63426b825263ea0a3e1917 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ab6805f01d63426b825263ea0a3e1917 2023-04-13 02:56:19,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ab6805f01d63426b825263ea0a3e1917, entries=150, sequenceid=668, filesize=11.9 K 2023-04-13 02:56:19,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 63bd78eb0ff0948d99f46c700cb54493 in 1246ms, sequenceid=668, compaction requested=false 2023-04-13 02:56:19,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:27,627 DEBUG [Thread-3697] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bcc330a to 127.0.0.1:62948 2023-04-13 02:56:27,627 DEBUG [Thread-3697] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:27,629 DEBUG [Thread-3701] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f1adb60 to 127.0.0.1:62948 2023-04-13 02:56:27,629 DEBUG [Thread-3701] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:27,706 DEBUG [Thread-3699] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23afc0f2 to 127.0.0.1:62948 2023-04-13 02:56:27,706 DEBUG [Thread-3699] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:27,721 DEBUG [Thread-3693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x089b6154 to 127.0.0.1:62948 2023-04-13 02:56:27,721 DEBUG [Thread-3693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 265 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4504 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4391 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4371 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4511 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(398): read 4366 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 02:56:27,721 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:56:27,721 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x041dc777 to 127.0.0.1:62948 2023-04-13 02:56:27,722 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:27,722 INFO [Listener at localhost/34379] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 02:56:27,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 02:56:27,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:27,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:27,725 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354587725"}]},"ts":"1681354587725"} 2023-04-13 02:56:27,726 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 02:56:27,730 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 02:56:27,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, UNASSIGN}] 2023-04-13 02:56:27,731 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, UNASSIGN 2023-04-13 02:56:27,732 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:27,732 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354587732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681354587732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681354587732"}]},"ts":"1681354587732"} 2023-04-13 02:56:27,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534}] 2023-04-13 02:56:27,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:27,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 63bd78eb0ff0948d99f46c700cb54493, disabling compactions & flushes 2023-04-13 02:56:27,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. after waiting 0 ms 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:27,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 63bd78eb0ff0948d99f46c700cb54493 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=A 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=B 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63bd78eb0ff0948d99f46c700cb54493, store=C 2023-04-13 02:56:27,885 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 02:56:27,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d39255855ef546a6b77a844c3ab46821 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d39255855ef546a6b77a844c3ab46821 2023-04-13 02:56:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:28,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=676, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/a76d47900ace46f39249bfb6cf57ed4b 2023-04-13 02:56:28,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/85bc716fa29443589a4c9ec5a23820f6 2023-04-13 02:56:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:28,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/703b2b6b1868461a94bde16cf6b4901f 2023-04-13 02:56:28,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/A/a76d47900ace46f39249bfb6cf57ed4b as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/a76d47900ace46f39249bfb6cf57ed4b 2023-04-13 02:56:28,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/a76d47900ace46f39249bfb6cf57ed4b, entries=150, sequenceid=676, filesize=25.4 K 2023-04-13 02:56:28,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/B/85bc716fa29443589a4c9ec5a23820f6 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/85bc716fa29443589a4c9ec5a23820f6 2023-04-13 02:56:28,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/85bc716fa29443589a4c9ec5a23820f6, entries=150, sequenceid=676, filesize=11.9 K 2023-04-13 02:56:28,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/.tmp/C/703b2b6b1868461a94bde16cf6b4901f as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/703b2b6b1868461a94bde16cf6b4901f 2023-04-13 02:56:28,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/703b2b6b1868461a94bde16cf6b4901f, entries=150, sequenceid=676, filesize=11.9 K 2023-04-13 02:56:28,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 63bd78eb0ff0948d99f46c700cb54493 in 841ms, sequenceid=676, compaction requested=true 2023-04-13 02:56:28,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/41012cc60d9642e98f2e0724e93fd0c2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/69970bea35cb4886ada5b5df40d98d7c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9d4a1cce24bb439c9f77da561ab1bf19, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/5ef0ccf61ad04a399de4b21351a44d56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/7d36889bc6194b63987538241649524d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2fb4f748631c4d59b4b7bd655e215a07, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9450ad520d8b46c4a66f458c9f6bb48f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/932df346dd5a4369aa73e0dc33107ff6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f65a740889aa4c929d1ad6370f1373fb, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7] to archive 2023-04-13 02:56:28,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:56:28,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2e11b4b63ac346b6a1f5fd5eb5d85fce 2023-04-13 02:56:28,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/c61119f03886486994d8e05a6a1aa270 2023-04-13 02:56:28,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/275aaea0c7d34c6999e6d910e98f1a32 2023-04-13 02:56:28,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/41012cc60d9642e98f2e0724e93fd0c2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/41012cc60d9642e98f2e0724e93fd0c2 2023-04-13 02:56:28,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/d842513e20d54db29661ce6e392a2ebf 2023-04-13 02:56:28,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b6f2d1c187ac469db2d16d12723f1ad7 2023-04-13 02:56:28,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/69970bea35cb4886ada5b5df40d98d7c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/69970bea35cb4886ada5b5df40d98d7c 2023-04-13 02:56:28,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/b1190c41cad748038adee6ec139fc2a4 2023-04-13 02:56:28,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ff5dfe91e66a46578b98c7b6df1d0df0 2023-04-13 02:56:28,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/97f7268f84c24eba8b5bf170ed5bc425 2023-04-13 02:56:28,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9d4a1cce24bb439c9f77da561ab1bf19 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9d4a1cce24bb439c9f77da561ab1bf19 2023-04-13 02:56:28,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ade9a8efdbbd4514ad876f26084d558b 2023-04-13 02:56:28,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/45f55e87be094ad794010004d6f39982 2023-04-13 02:56:28,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/161ba1be88a04f54bb6edd27b8c72042 2023-04-13 02:56:28,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/8d2b467769d446b0ba6f8ee7082f765c 2023-04-13 02:56:28,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/50e4139d2bec4305b8c6d8b3a1567867 2023-04-13 02:56:28,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/5ef0ccf61ad04a399de4b21351a44d56 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/5ef0ccf61ad04a399de4b21351a44d56 2023-04-13 02:56:28,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/3c4fa7c2a4264361a6524c093bea1884 2023-04-13 02:56:28,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/7d36889bc6194b63987538241649524d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/7d36889bc6194b63987538241649524d 2023-04-13 02:56:28,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/822009fbbaec4b2a883349975dfba7d5 2023-04-13 02:56:28,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/07de2cba946e4f30914981d377a08b02 2023-04-13 02:56:28,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/869a6550010f4bb7aee18f246878f0e1 2023-04-13 02:56:28,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2fb4f748631c4d59b4b7bd655e215a07 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2fb4f748631c4d59b4b7bd655e215a07 2023-04-13 02:56:28,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f8203d7eb17c411e9b0b3e09bae41967 2023-04-13 02:56:28,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/33592e96fb274caf885ddf71373c62c9 2023-04-13 02:56:28,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/ca0e74718bfc459cb788941ef71a2b01 2023-04-13 02:56:28,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9450ad520d8b46c4a66f458c9f6bb48f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/9450ad520d8b46c4a66f458c9f6bb48f 2023-04-13 02:56:28,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2afccbca15784144b27c0a43197e5410 2023-04-13 02:56:28,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/421182b9d6e64405b36bf94f94183343 2023-04-13 02:56:28,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f1ea37a9ac544bb1bc17634074cdf7bf 2023-04-13 02:56:28,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/932df346dd5a4369aa73e0dc33107ff6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/932df346dd5a4369aa73e0dc33107ff6 2023-04-13 02:56:28,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/cc8a752adcac433886c363ceb48d8ba9 2023-04-13 02:56:28,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4998b1555cce48369896924064b7a399 2023-04-13 02:56:28,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f65a740889aa4c929d1ad6370f1373fb to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/f65a740889aa4c929d1ad6370f1373fb 2023-04-13 02:56:28,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/4e3b7824b36a40acae4b9b04c7891f02 2023-04-13 02:56:28,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/dd8ff15987634d2d802a00614a6d2674 2023-04-13 02:56:28,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/2bdc3a56393243d09e25a65248db97a7 2023-04-13 02:56:28,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7e18137c4bb44b9681c35934bf2df064, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/d56750e0889c4bdf83dcf91b0cb66a6c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9cb763b352e140cb9f320ea37c31136f, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7905abb9df384eb2ba3f39c91b43a12a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/33b4ca87a9e14cc5b505848505114303, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3d9f91fb34a84257a3d672451c6b3100, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/60b87f9fa77045ed8b9ee8924ec4ab36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2d6c6f24c36c4720be3ca13e95ee772a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/894c05120f944b2ca91ab8c814bbc2c7, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3] to archive 2023-04-13 02:56:28,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:56:28,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/47a08f02d6584e6cbb8dd2a453777abc 2023-04-13 02:56:28,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/663f96213a36430e8f97e25c0fcef9c1 2023-04-13 02:56:28,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7e18137c4bb44b9681c35934bf2df064 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7e18137c4bb44b9681c35934bf2df064 2023-04-13 02:56:28,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7297d8d252804c42879177de28b9c13f 2023-04-13 02:56:28,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/22f819c28f6d4caa95cb2da916f9965f 2023-04-13 02:56:28,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/d56750e0889c4bdf83dcf91b0cb66a6c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/d56750e0889c4bdf83dcf91b0cb66a6c 2023-04-13 02:56:28,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a678a9062a2f4560b3c6a1131d08424b 2023-04-13 02:56:28,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3f2f53d0a5d344c68877935ee6da0e74 2023-04-13 02:56:28,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/40f74d0669874931a583bd6f38cf47b8 2023-04-13 02:56:28,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9cb763b352e140cb9f320ea37c31136f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9cb763b352e140cb9f320ea37c31136f 2023-04-13 02:56:28,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/19e95f6f5dff48898ba3dfc6b97e016b 2023-04-13 02:56:28,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/9c86830e91ac423c97acd0bf8f21487d 2023-04-13 02:56:28,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/5198af817b2840ad83fe8e98a2e81abf 2023-04-13 02:56:28,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/b80634f3280f45d08e64b27456ad0532 2023-04-13 02:56:28,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/560286be63544b4e95f6e5e90250d3ce 2023-04-13 02:56:28,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7905abb9df384eb2ba3f39c91b43a12a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/7905abb9df384eb2ba3f39c91b43a12a 2023-04-13 02:56:28,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/93b4b8d816484496b8d49f1b5c451d9d 2023-04-13 02:56:28,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1e1d3bafb24549d689c4afdcffe658b5 2023-04-13 02:56:28,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/33b4ca87a9e14cc5b505848505114303 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/33b4ca87a9e14cc5b505848505114303 2023-04-13 02:56:28,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/6ff591c97dc54b119d7a9fe3aa18e807 2023-04-13 02:56:28,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/96400567d3cf41da836555f5eea00d3c 2023-04-13 02:56:28,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3d9f91fb34a84257a3d672451c6b3100 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/3d9f91fb34a84257a3d672451c6b3100 2023-04-13 02:56:28,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cb74f9a33ad04d078bd10949ed61fd68 2023-04-13 02:56:28,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/1f65759a8c344692b16f032706782049 2023-04-13 02:56:28,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/4152b009fac54f259b6a0394ac44930d 2023-04-13 02:56:28,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/60b87f9fa77045ed8b9ee8924ec4ab36 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/60b87f9fa77045ed8b9ee8924ec4ab36 2023-04-13 02:56:28,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/8325c9aeda7849d8bc7b6aa8c7763a08 2023-04-13 02:56:28,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/489e5aaa64b7496a83f7627e46cfb508 2023-04-13 02:56:28,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/0872cafcc9444caa9be09c0097ae5a90 2023-04-13 02:56:28,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2d6c6f24c36c4720be3ca13e95ee772a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2d6c6f24c36c4720be3ca13e95ee772a 2023-04-13 02:56:28,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/f0d6a9ecefc34ea7be1a5e19ef59e879 2023-04-13 02:56:28,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/cc3cfc0e9ac24f819f5b93455574ecda 2023-04-13 02:56:28,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/49da5ecba1644be4984d865d5a499998 2023-04-13 02:56:28,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/894c05120f944b2ca91ab8c814bbc2c7 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/894c05120f944b2ca91ab8c814bbc2c7 2023-04-13 02:56:28,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/18358ed45719454a8e9c8a26ed49efa2 2023-04-13 02:56:28,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/e6c0d57773544bc5844a4bdfd0007378 2023-04-13 02:56:28,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/c23f35b2f9674115b598d66a08ab00d3 2023-04-13 02:56:28,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/8f68234121654da9b0e1c0d13cc38a3a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/3ba5588c6d4a4031bb50a3569bb34dd9, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0f4af183235943be8827d9f90e85915e, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c37396194694491a9223c26015f4adbc, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f6eb35735d294c29bdd717a17f95a818, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/585f612cee24408c8395f10e3f034baa, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/65858cfbb8b14e01810f49f1901a4525, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/82b45424697847bdbb4f23f31a7da072, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1] to archive 2023-04-13 02:56:28,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 02:56:28,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/2b695009414a46ae89b37f7f2546dd6d 2023-04-13 02:56:28,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/cbfd57be96434b55be7f3b661efbcc3b 2023-04-13 02:56:28,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/8f68234121654da9b0e1c0d13cc38a3a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/8f68234121654da9b0e1c0d13cc38a3a 2023-04-13 02:56:28,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/09e5c2d24a92464ca08f0a62ed945660 2023-04-13 02:56:28,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/67727d9c63724682b04854628d81c584 2023-04-13 02:56:28,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/3ba5588c6d4a4031bb50a3569bb34dd9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/3ba5588c6d4a4031bb50a3569bb34dd9 2023-04-13 02:56:28,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5442f28805144b66b1f07b4daa547865 2023-04-13 02:56:28,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b23017dd4576492a9830fda855d3bd36 2023-04-13 02:56:28,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/baa36d94c82946c0939cb74f2e2d5993 2023-04-13 02:56:28,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/31f98ef8b2e34b00a3fa0c90c0fdb4bd 2023-04-13 02:56:28,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/33a494a865154e258f5d8a3101c75b58 2023-04-13 02:56:28,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c7c22131eddb4e28bc7b9f2f303d8ed6 2023-04-13 02:56:28,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/321fa12a698847fa85260cc090d5e607 2023-04-13 02:56:28,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/b5a1ce69785e4488bc3f29f6c3922029 2023-04-13 02:56:28,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c4383c271deb42a391a6e60c303520b5 2023-04-13 02:56:28,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0f4af183235943be8827d9f90e85915e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0f4af183235943be8827d9f90e85915e 2023-04-13 02:56:28,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/0d7cf1cc87884fc79d92ca4a0a58adaa 2023-04-13 02:56:28,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/098987fb027c455cba612ab918eefdab 2023-04-13 02:56:28,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c37396194694491a9223c26015f4adbc to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/c37396194694491a9223c26015f4adbc 2023-04-13 02:56:28,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/55d7be54fc9544e89c4dafa53d812c18 2023-04-13 02:56:28,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/5cd089e85d684c118ebcaf312810e823 2023-04-13 02:56:28,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f6eb35735d294c29bdd717a17f95a818 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f6eb35735d294c29bdd717a17f95a818 2023-04-13 02:56:28,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/750b62ffaa0f4da88e8a0cbf6c65cf5a 2023-04-13 02:56:28,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f830e7ebde2242ef887aba8632f94b72 2023-04-13 02:56:28,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/f0621aabfc7b4ccc860e1081145b8b56 2023-04-13 02:56:28,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/585f612cee24408c8395f10e3f034baa to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/585f612cee24408c8395f10e3f034baa 2023-04-13 02:56:28,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e5129d5ee9348b29dd25c745a805b82 2023-04-13 02:56:28,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/fb5c0b2c29c84a068a1c3152b616a3a0 2023-04-13 02:56:28,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/820f41ee1e5c4e90b509179634dc6b0d 2023-04-13 02:56:28,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/65858cfbb8b14e01810f49f1901a4525 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/65858cfbb8b14e01810f49f1901a4525 2023-04-13 02:56:28,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/32f05e000a474ced90f361dda97af275 2023-04-13 02:56:28,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ae15b071262e4ff4a3749f8bf8a63bbf 2023-04-13 02:56:28,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/34719fb205144208b2f48f95bdb2ca46 2023-04-13 02:56:28,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/82b45424697847bdbb4f23f31a7da072 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/82b45424697847bdbb4f23f31a7da072 2023-04-13 02:56:28,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/890eb5786e7645ccb4722861901e0071 2023-04-13 02:56:28,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/9d8ef84501fe45af885e7f5d1d3609bd 2023-04-13 02:56:28,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/41d1e2d816354428831352993ba443d1 2023-04-13 02:56:28,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:28,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits/679.seqid, newMaxSeqId=679, maxSeqId=4 2023-04-13 02:56:28,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493. 2023-04-13 02:56:28,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 63bd78eb0ff0948d99f46c700cb54493: 2023-04-13 02:56:28,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:28,836 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=63bd78eb0ff0948d99f46c700cb54493, regionState=CLOSED 2023-04-13 02:56:28,836 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681354588836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681354588836"}]},"ts":"1681354588836"} 2023-04-13 02:56:28,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-13 02:56:28,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 63bd78eb0ff0948d99f46c700cb54493, server=jenkins-hbase4.apache.org,34919,1681354416534 in 1.1040 sec 2023-04-13 02:56:28,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-13 02:56:28,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63bd78eb0ff0948d99f46c700cb54493, UNASSIGN in 1.1070 sec 2023-04-13 02:56:28,839 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681354588839"}]},"ts":"1681354588839"} 2023-04-13 02:56:28,840 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 02:56:28,841 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 02:56:28,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1190 sec 2023-04-13 02:56:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 02:56:29,830 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-13 02:56:29,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 02:56:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,833 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,833 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 02:56:29,838 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:29,839 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C, FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits] 2023-04-13 02:56:29,842 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/a76d47900ace46f39249bfb6cf57ed4b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/a76d47900ace46f39249bfb6cf57ed4b 2023-04-13 02:56:29,843 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bacdb3ccab494385b0701bf58a101330 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bacdb3ccab494385b0701bf58a101330 2023-04-13 02:56:29,844 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bb095a6df472417babaa1823ad456161 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/A/bb095a6df472417babaa1823ad456161 2023-04-13 02:56:29,846 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2f880b99fe1e4e2b82048fe5acf0098e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/2f880b99fe1e4e2b82048fe5acf0098e 2023-04-13 02:56:29,847 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/85bc716fa29443589a4c9ec5a23820f6 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/85bc716fa29443589a4c9ec5a23820f6 2023-04-13 02:56:29,848 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a3f92a8b66684f6fb676c0211c3dddfe to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/B/a3f92a8b66684f6fb676c0211c3dddfe 2023-04-13 02:56:29,850 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/703b2b6b1868461a94bde16cf6b4901f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/703b2b6b1868461a94bde16cf6b4901f 2023-04-13 02:56:29,851 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e6f459369c84276b459cae28afe05ce to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/7e6f459369c84276b459cae28afe05ce 2023-04-13 02:56:29,852 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ab6805f01d63426b825263ea0a3e1917 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/C/ab6805f01d63426b825263ea0a3e1917 2023-04-13 02:56:29,854 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits/679.seqid to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493/recovered.edits/679.seqid 2023-04-13 02:56:29,854 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/.tmp/data/default/TestAcidGuarantees/63bd78eb0ff0948d99f46c700cb54493 2023-04-13 02:56:29,854 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 02:56:29,855 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:56:29,855 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 02:56:29,857 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130b83b8545bb74accb87e5714b23b1f12 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130b83b8545bb74accb87e5714b23b1f12 2023-04-13 02:56:29,858 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041320c69db268f74effa24e0c73f5752b0e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041320c69db268f74effa24e0c73f5752b0e 2023-04-13 02:56:29,859 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fe23e16c9e45249333f73882eba9a9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041324fe23e16c9e45249333f73882eba9a9 2023-04-13 02:56:29,859 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326847f8c0e7342228cdd73aa8beb7e21 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041326847f8c0e7342228cdd73aa8beb7e21 2023-04-13 02:56:29,860 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133bca2b07328b447aababf8cbb886eb98 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133bca2b07328b447aababf8cbb886eb98 2023-04-13 02:56:29,861 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041340ddcca4f1924689bf61169b1c611dec to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041340ddcca4f1924689bf61169b1c611dec 2023-04-13 02:56:29,862 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344fad3cb46544cc1a8fe4c7733663135 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041344fad3cb46544cc1a8fe4c7733663135 2023-04-13 02:56:29,863 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413519de5d032934622aa6f4d6d6457c337 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413519de5d032934622aa6f4d6d6457c337 2023-04-13 02:56:29,864 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413580f58af3a8e4dc6bcf0aadc6ed7ba3d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413580f58af3a8e4dc6bcf0aadc6ed7ba3d 2023-04-13 02:56:29,865 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c540ecae014479dac31d81929d4efe9 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c540ecae014479dac31d81929d4efe9 2023-04-13 02:56:29,866 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413613f927793b24ae18c8ffbbf1ca41bf3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413613f927793b24ae18c8ffbbf1ca41bf3 2023-04-13 02:56:29,866 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136450a0d0bea347eeb1f7c6aac4d92851 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136450a0d0bea347eeb1f7c6aac4d92851 2023-04-13 02:56:29,867 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041368fcaa093ab84f0cbcba13f3ebc76837 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041368fcaa093ab84f0cbcba13f3ebc76837 2023-04-13 02:56:29,868 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137140afbbd0424d7fbdee7e62c08d1b0f to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137140afbbd0424d7fbdee7e62c08d1b0f 2023-04-13 02:56:29,869 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413781dd5cd1fa241d799b7baef8845721c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413781dd5cd1fa241d799b7baef8845721c 2023-04-13 02:56:29,870 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137a3155ddda7a430f865f19b6fc1d55b1 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137a3155ddda7a430f865f19b6fc1d55b1 2023-04-13 02:56:29,871 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137d19d2fa658447a3ac83cedf03c6d5ba to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137d19d2fa658447a3ac83cedf03c6d5ba 2023-04-13 02:56:29,871 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041384b820e2b7dd49bd9d9065bfa5c0bf0d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041384b820e2b7dd49bd9d9065bfa5c0bf0d 2023-04-13 02:56:29,872 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041392f15237f48d40e7976459d56e1d7ad5 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041392f15237f48d40e7976459d56e1d7ad5 2023-04-13 02:56:29,873 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041396a56a68b38346e28910900685d6be33 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041396a56a68b38346e28910900685d6be33 2023-04-13 02:56:29,874 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413af150b67bdd64b1080f691fbf9bbffd3 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413af150b67bdd64b1080f691fbf9bbffd3 2023-04-13 02:56:29,875 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c643d0c22c0d4f3db3b4d0f4bab9ae5b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c643d0c22c0d4f3db3b4d0f4bab9ae5b 2023-04-13 02:56:29,876 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d39255855ef546a6b77a844c3ab46821 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d39255855ef546a6b77a844c3ab46821 2023-04-13 02:56:29,877 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4b47bb423cf4666b371b27bf4dcd961 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d4b47bb423cf4666b371b27bf4dcd961 2023-04-13 02:56:29,877 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2d361a09eba462a8cc961a0dbbbcb19 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e2d361a09eba462a8cc961a0dbbbcb19 2023-04-13 02:56:29,878 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed15f0ad916b448d946a86586d5d0a1b to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed15f0ad916b448d946a86586d5d0a1b 2023-04-13 02:56:29,879 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed8fcbdf780844cf96450d59674db47e to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ed8fcbdf780844cf96450d59674db47e 2023-04-13 02:56:29,880 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413efd7ba6990de419fa76d956ab23b237d to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413efd7ba6990de419fa76d956ab23b237d 2023-04-13 02:56:29,881 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f430dcdf4bf3462f89ad1bcfb54f751c to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f430dcdf4bf3462f89ad1bcfb54f751c 2023-04-13 02:56:29,882 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fdd17dfad04f4c8486ef0a47b7166900 to hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fdd17dfad04f4c8486ef0a47b7166900 2023-04-13 02:56:29,882 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 02:56:29,884 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,885 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 02:56:29,886 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 02:56:29,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 02:56:29,887 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681354589887"}]},"ts":"9223372036854775807"} 2023-04-13 02:56:29,888 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 02:56:29,888 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 63bd78eb0ff0948d99f46c700cb54493, NAME => 'TestAcidGuarantees,,1681354555555.63bd78eb0ff0948d99f46c700cb54493.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 02:56:29,889 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 02:56:29,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681354589889"}]},"ts":"9223372036854775807"} 2023-04-13 02:56:29,890 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 02:56:29,891 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 02:56:29,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-04-13 02:56:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 02:56:29,934 INFO [Listener at localhost/34379] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-13 02:56:29,946 INFO [Listener at localhost/34379] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 228), OpenFileDescriptor=523 (was 623), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 470), ProcessCount=173 (was 173), AvailableMemoryMB=1246 (was 1117) - AvailableMemoryMB LEAK? - 2023-04-13 02:56:29,946 INFO [Listener at localhost/34379] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-13 02:56:29,946 INFO [Listener at localhost/34379] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 02:56:29,946 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45b2cf6e to 127.0.0.1:62948 2023-04-13 02:56:29,946 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:29,947 DEBUG [Listener at localhost/34379] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-13 02:56:29,947 DEBUG [Listener at localhost/34379] util.JVMClusterUtil(257): Found active master hash=1143794569, stopped=false 2023-04-13 02:56:29,947 INFO [Listener at localhost/34379] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:56:29,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 02:56:29,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 02:56:29,948 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:56:29,948 INFO [Listener at localhost/34379] procedure2.ProcedureExecutor(629): Stopping 2023-04-13 02:56:29,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 02:56:29,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 02:56:29,949 DEBUG [Listener at localhost/34379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01d21cc5 to 127.0.0.1:62948 2023-04-13 02:56:29,949 DEBUG [Listener at localhost/34379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:29,949 INFO [Listener at localhost/34379] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34919,1681354416534' ***** 2023-04-13 02:56:29,949 INFO [Listener at localhost/34379] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-13 02:56:29,950 INFO [RS:0;jenkins-hbase4:34919] regionserver.HeapMemoryManager(220): Stopping 2023-04-13 02:56:29,950 INFO [RS:0;jenkins-hbase4:34919] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-13 02:56:29,950 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-13 02:56:29,950 INFO [RS:0;jenkins-hbase4:34919] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-13 02:56:29,950 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(3304): Received CLOSE for 1926898cb36187246d86e87ddb72126d 2023-04-13 02:56:29,951 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:29,951 DEBUG [RS:0;jenkins-hbase4:34919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ac91733 to 127.0.0.1:62948 2023-04-13 02:56:29,951 DEBUG [RS:0;jenkins-hbase4:34919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:29,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1926898cb36187246d86e87ddb72126d, disabling compactions & flushes 2023-04-13 02:56:29,951 INFO [RS:0;jenkins-hbase4:34919] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-13 02:56:29,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:56:29,951 INFO [RS:0;jenkins-hbase4:34919] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-13 02:56:29,951 INFO [RS:0;jenkins-hbase4:34919] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-13 02:56:29,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:56:29,951 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-13 02:56:29,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. after waiting 0 ms 2023-04-13 02:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:56:29,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1926898cb36187246d86e87ddb72126d 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-13 02:56:29,952 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-13 02:56:29,952 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 1926898cb36187246d86e87ddb72126d=hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d.} 2023-04-13 02:56:29,952 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 02:56:29,952 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 02:56:29,952 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 02:56:29,952 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 02:56:29,952 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 02:56:29,953 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-13 02:56:29,954 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1505): Waiting on 1588230740, 1926898cb36187246d86e87ddb72126d 2023-04-13 02:56:29,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/.tmp/info/05a85d2c0fcc4e8b865e53fa54e98d20 2023-04-13 02:56:29,960 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/info/194b39962f2f4ae6a6939106adf68a14 2023-04-13 02:56:29,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/.tmp/info/05a85d2c0fcc4e8b865e53fa54e98d20 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/info/05a85d2c0fcc4e8b865e53fa54e98d20 2023-04-13 02:56:29,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 194b39962f2f4ae6a6939106adf68a14 2023-04-13 02:56:29,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/info/05a85d2c0fcc4e8b865e53fa54e98d20, entries=2, sequenceid=6, filesize=4.8 K 2023-04-13 02:56:29,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1926898cb36187246d86e87ddb72126d in 15ms, sequenceid=6, compaction requested=false 2023-04-13 02:56:29,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/namespace/1926898cb36187246d86e87ddb72126d/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-13 02:56:29,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:56:29,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1926898cb36187246d86e87ddb72126d: 2023-04-13 02:56:29,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681354418907.1926898cb36187246d86e87ddb72126d. 2023-04-13 02:56:30,029 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-13 02:56:30,154 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 02:56:30,354 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 02:56:30,377 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/rep_barrier/cc82c5a6181c43ef922810db6e96e926 2023-04-13 02:56:30,380 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc82c5a6181c43ef922810db6e96e926 2023-04-13 02:56:30,555 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 02:56:30,755 DEBUG [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 02:56:30,789 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/table/2e525ea6a0b74fcc9f8a8a55686b9efb 2023-04-13 02:56:30,792 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2e525ea6a0b74fcc9f8a8a55686b9efb 2023-04-13 02:56:30,792 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/info/194b39962f2f4ae6a6939106adf68a14 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/info/194b39962f2f4ae6a6939106adf68a14 2023-04-13 02:56:30,795 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 194b39962f2f4ae6a6939106adf68a14 2023-04-13 02:56:30,795 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/info/194b39962f2f4ae6a6939106adf68a14, entries=22, sequenceid=93, filesize=7.1 K 2023-04-13 02:56:30,797 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/rep_barrier/cc82c5a6181c43ef922810db6e96e926 as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/rep_barrier/cc82c5a6181c43ef922810db6e96e926 2023-04-13 02:56:30,799 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc82c5a6181c43ef922810db6e96e926 2023-04-13 02:56:30,799 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/rep_barrier/cc82c5a6181c43ef922810db6e96e926, entries=6, sequenceid=93, filesize=5.5 K 2023-04-13 02:56:30,800 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/.tmp/table/2e525ea6a0b74fcc9f8a8a55686b9efb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/table/2e525ea6a0b74fcc9f8a8a55686b9efb 2023-04-13 02:56:30,802 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2e525ea6a0b74fcc9f8a8a55686b9efb 2023-04-13 02:56:30,802 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/table/2e525ea6a0b74fcc9f8a8a55686b9efb, entries=9, sequenceid=93, filesize=5.5 K 2023-04-13 02:56:30,803 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 851ms, sequenceid=93, compaction requested=false 2023-04-13 02:56:30,807 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-13 02:56:30,808 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 02:56:30,809 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 02:56:30,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 02:56:30,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-13 02:56:30,857 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-13 02:56:30,857 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-13 02:56:30,955 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34919,1681354416534; all regions closed. 2023-04-13 02:56:30,963 DEBUG [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/oldWALs 2023-04-13 02:56:30,963 INFO [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34919%2C1681354416534.meta:.meta(num 1681354418624) 2023-04-13 02:56:30,967 DEBUG [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/oldWALs 2023-04-13 02:56:30,967 INFO [RS:0;jenkins-hbase4:34919] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34919%2C1681354416534:(num 1681354418011) 2023-04-13 02:56:30,967 DEBUG [RS:0;jenkins-hbase4:34919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:30,967 INFO [RS:0;jenkins-hbase4:34919] regionserver.LeaseManager(133): Closed leases 2023-04-13 02:56:30,968 INFO [RS:0;jenkins-hbase4:34919] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-13 02:56:30,968 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 02:56:30,970 INFO [RS:0;jenkins-hbase4:34919] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34919 2023-04-13 02:56:30,978 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 02:56:30,978 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34919,1681354416534 2023-04-13 02:56:30,978 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a55d954 rejected from java.util.concurrent.ThreadPoolExecutor@7e6a42c6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2504] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:30,979 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 02:56:30,979 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34919,1681354416534] 2023-04-13 02:56:30,979 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5e52822d rejected from java.util.concurrent.ThreadPoolExecutor@7e6a42c6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2504] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:30,980 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34919,1681354416534; numProcessing=1 2023-04-13 02:56:30,981 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34919,1681354416534 already deleted, retry=false 2023-04-13 02:56:30,981 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34919,1681354416534 expired; onlineServers=0 2023-04-13 02:56:30,981 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37305,1681354415362' ***** 2023-04-13 02:56:30,981 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-13 02:56:30,981 DEBUG [M:0;jenkins-hbase4:37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@511e1b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 02:56:30,981 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37305,1681354415362 2023-04-13 02:56:30,981 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37305,1681354415362; all regions closed. 2023-04-13 02:56:30,981 DEBUG [M:0;jenkins-hbase4:37305] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 02:56:30,981 DEBUG [M:0;jenkins-hbase4:37305] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-13 02:56:30,981 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-13 02:56:30,982 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681354417735] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681354417735,5,FailOnTimeoutGroup] 2023-04-13 02:56:30,982 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681354417736] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681354417736,5,FailOnTimeoutGroup] 2023-04-13 02:56:30,982 DEBUG [M:0;jenkins-hbase4:37305] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-13 02:56:30,982 INFO [M:0;jenkins-hbase4:37305] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-13 02:56:30,982 INFO [M:0;jenkins-hbase4:37305] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-13 02:56:30,982 INFO [M:0;jenkins-hbase4:37305] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-13 02:56:30,983 DEBUG [M:0;jenkins-hbase4:37305] master.HMaster(1512): Stopping service threads 2023-04-13 02:56:30,983 INFO [M:0;jenkins-hbase4:37305] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-13 02:56:30,983 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-13 02:56:30,983 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 02:56:30,983 ERROR [M:0;jenkins-hbase4:37305] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (2115436087) connection to localhost/127.0.0.1:37981 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-13 02:56:30,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 02:56:30,983 INFO [M:0;jenkins-hbase4:37305] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-13 02:56:30,983 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-13 02:56:30,984 DEBUG [M:0;jenkins-hbase4:37305] zookeeper.ZKUtil(398): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-13 02:56:30,984 WARN [M:0;jenkins-hbase4:37305] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-13 02:56:30,984 INFO [M:0;jenkins-hbase4:37305] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-13 02:56:30,984 INFO [M:0;jenkins-hbase4:37305] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-13 02:56:30,985 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 02:56:30,985 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:56:30,985 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:56:30,985 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 02:56:30,985 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:56:30,985 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.84 KB heapSize=393.61 KB 2023-04-13 02:56:31,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 02:56:31,081 INFO [RS:0;jenkins-hbase4:34919] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34919,1681354416534; zookeeper connection closed. 2023-04-13 02:56:31,081 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2649602d rejected from java.util.concurrent.ThreadPoolExecutor@7e6a42c6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2504] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:31,081 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): regionserver:34919-0x1017b7043800001, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 02:56:31,081 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@11ce8e8f rejected from java.util.concurrent.ThreadPoolExecutor@7e6a42c6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2504] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:31,081 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d446f3a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d446f3a 2023-04-13 02:56:31,082 INFO [Listener at localhost/34379] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-13 02:56:31,393 INFO [M:0;jenkins-hbase4:37305] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.84 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e88bb4c4dfaa4aaeab8e1aa07311edbb 2023-04-13 02:56:31,397 INFO [M:0;jenkins-hbase4:37305] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e88bb4c4dfaa4aaeab8e1aa07311edbb 2023-04-13 02:56:31,397 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e88bb4c4dfaa4aaeab8e1aa07311edbb as hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e88bb4c4dfaa4aaeab8e1aa07311edbb 2023-04-13 02:56:31,400 INFO [M:0;jenkins-hbase4:37305] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e88bb4c4dfaa4aaeab8e1aa07311edbb 2023-04-13 02:56:31,401 INFO [M:0;jenkins-hbase4:37305] regionserver.HStore(1080): Added hdfs://localhost:37981/user/jenkins/test-data/d7c50f70-d361-9821-c9db-89f575ea2a55/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e88bb4c4dfaa4aaeab8e1aa07311edbb, entries=65, sequenceid=578, filesize=24.7 K 2023-04-13 02:56:31,401 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegion(2948): Finished flush of dataSize ~344.84 KB/353113, heapSize ~393.59 KB/403040, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 416ms, sequenceid=578, compaction requested=false 2023-04-13 02:56:31,402 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 02:56:31,402 DEBUG [M:0;jenkins-hbase4:37305] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 02:56:31,406 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 02:56:31,406 INFO [M:0;jenkins-hbase4:37305] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-13 02:56:31,407 INFO [M:0;jenkins-hbase4:37305] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37305 2023-04-13 02:56:31,409 DEBUG [M:0;jenkins-hbase4:37305] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37305,1681354415362 already deleted, retry=false 2023-04-13 02:56:31,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 02:56:31,511 INFO [M:0;jenkins-hbase4:37305] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37305,1681354415362; zookeeper connection closed. 2023-04-13 02:56:31,511 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@398ec836 rejected from java.util.concurrent.ThreadPoolExecutor@358b7cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4186] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:31,511 DEBUG [Listener at localhost/34379-EventThread] zookeeper.ZKWatcher(600): master:37305-0x1017b7043800000, quorum=127.0.0.1:62948, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 02:56:31,511 ERROR [Listener at localhost/34379-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@11fcfd83 rejected from java.util.concurrent.ThreadPoolExecutor@358b7cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4186] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 02:56:31,514 WARN [Listener at localhost/34379] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-13 02:56:31,517 INFO [Listener at localhost/34379] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 02:56:31,625 WARN [BP-878814799-172.31.14.131-1681354412046 heartbeating to localhost/127.0.0.1:37981] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-13 02:56:31,625 WARN [BP-878814799-172.31.14.131-1681354412046 heartbeating to localhost/127.0.0.1:37981] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-878814799-172.31.14.131-1681354412046 (Datanode Uuid c27897a0-e461-4578-8075-79a476069002) service to localhost/127.0.0.1:37981 2023-04-13 02:56:31,629 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25/dfs/data/data1/current/BP-878814799-172.31.14.131-1681354412046] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 02:56:31,630 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/588dc7b4-3982-ceb0-8550-9b26a419d6fc/cluster_1fa82e51-8645-f2a6-b553-2d72ed7a6e25/dfs/data/data2/current/BP-878814799-172.31.14.131-1681354412046] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 02:56:31,666 INFO [Listener at localhost/34379] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 02:56:31,836 INFO [Listener at localhost/34379] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-13 02:56:32,021 INFO [Listener at localhost/34379] hbase.HBaseTestingUtility(1293): Minicluster is down